skip to main content
10.1145/1148170.1148262acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

Statistical precision of information retrieval evaluation

Published: 06 August 2006 Publication History

Abstract

We introduce and validate bootstrap techniques to compute confidence intervals that quantify the effect of test-collection variability on average precision (AP) and mean average precision (MAP) IR effectiveness measures. We consider the test collection in IR evaluation to be a representative of a population of materially similar collections, whose documents are drawn from an infinite pool with similar characteristics. Our model accurately predicts the degree of concordance between system results on randomly selected halves of the TREC-6 ad hoc corpus. We advance a framework for statistical evaluation that uses the same general framework to model other sources of chance variation as a source of input for meta-analysis techniques.

References

[1]
Buckley, C., and Voorhees, E. M. Evaluating evaluation measure stability. In SIGIR Conference 2000 (Athens, Greece, 2000).
[2]
Cormack, G. V., Palmer, C. R., and Clarke, C. L. A. Efficient construction of large test collections. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
[3]
Efron, B., and Tsibirani, R. J. An Introduction to the Bootstrap. Chapman and Hall, New York, 1994.
[4]
Fisher, R. A. Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society 22 (1925), 700--725.
[5]
Glass, G. V. Meta-analysis at 25. http://glass.ed.asu.edu/gene/papers/meta25.html, 2000.
[6]
Hull, D. A. Using statistical testing in the evaluation of retrieval experiments. In Research and Development in Information Retrieval (1993), pp. 329--338.
[7]
Lenhard, J. Models and statistical inference: The controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science (2006).
[8]
Rothman, K. J., and Greenland, S. Modern Epidemiology. Lippincott Williams & Wilkins, 1998.
[9]
Sanderson, M., and Johno, H. Test collections with no system pooling. In SIGIR Conference 2004 (Sheffield, UK, 2004).
[10]
Sanderson, M., and Zobel, J. Information retrieval evaluation: Effort, sensitivity, and reliability. In SIGIR Conference 2005 (Salvador, Brazil, 2005).
[11]
Savoy, J. Statistical inference in retrieval effectiveness evaluation. Information Processing and Management 33, 4 (1997), 495--512.
[12]
Tague-Sutcliffe, J. The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28, 4 (1992), 467--490.
[13]
Tague-Sutcliffe, J., and Blustein, J. A statistical analysis of the TREC-3 data. In Proceedings of TREC-3, The Third Information Retrieval Conference (1994), pp. 385--398.
[14]
Voorhees, E., and Harman, D. Overview of the Sixth Text REtrieval Conference (TREC-6). In 6th Text REtrieval Conference (Gaithersburg, MD, 1997).
[15]
Voorhees, E. M. Variations in relevance judgements and the measurement of retrieval effectiveness. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
[16]
Voorhees, E. M. Overview of the TREC-2004 robust track. In 13th Text REtrieval Conference (Gaithersburg, MD, 2004).
[17]
Voorhees, E. M., and Buckley, C. The effect of topic set size on retrieval experiment error. In SIGIR Conference 2002 (Tampere, Finland, 2002).
[18]
Voorhees, E. M., and Buckley, C. Retrieval evaluation with incomplete information. In SIGIR Conference 2004 (Sheffield, UK, 2004).
[19]
Voorhees, E. M., and Harman, D. K., Eds. TREC - Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
[20]
Zobel, J. How reliable are the results of large-scale information retrieval experiments? In SIGIR Conference 1998 (Melbourne, Australia, 1998).

Cited By

View all
  • (2024)Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I.Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671883(2307-2317)Online publication date: 25-Aug-2024
  • (2024)Evaluation metrics and statistical tests for machine learningScientific Reports10.1038/s41598-024-56706-x14:1Online publication date: 13-Mar-2024
  • (2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
  • Show More Cited By

Index Terms

  1. Statistical precision of information retrieval evaluation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
    August 2006
    768 pages
    ISBN:1595933697
    DOI:10.1145/1148170
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 August 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. bootstrap
    2. confidence interval
    3. precision

    Qualifiers

    • Article

    Conference

    SIGIR06
    Sponsor:
    SIGIR06: The 29th Annual International SIGIR Conference
    August 6 - 11, 2006
    Washington, Seattle, USA

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)54
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I.Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671883(2307-2317)Online publication date: 25-Aug-2024
    • (2024)Evaluation metrics and statistical tests for machine learningScientific Reports10.1038/s41598-024-56706-x14:1Online publication date: 13-Mar-2024
    • (2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
    • (2023)How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication MethodsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614916(1960-1970)Online publication date: 21-Oct-2023
    • (2023)When Measurement MisleadsACM SIGIR Forum10.1145/3582524.358254056:1(1-20)Online publication date: 27-Jan-2023
    • (2023)Evaluation of Cross-Lingual Bug Localization: Two Industrial Cases2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00063(495-499)Online publication date: 1-Oct-2023
    • (2023)Techniques, datasets, evaluation metrics and future directions of a question answering systemKnowledge and Information Systems10.1007/s10115-023-02019-w66:4(2235-2268)Online publication date: 22-Dec-2023
    • (2023)Bootstrapped nDCG Estimation in the Presence of Unjudged DocumentsAdvances in Information Retrieval10.1007/978-3-031-28244-7_20(313-329)Online publication date: 17-Mar-2023
    • (2022)A New Adaptive Indexing for Real-Time Web SearchInternational Journal of Intelligent Information Technologies10.4018/IJIIT.30958018:1(1-19)Online publication date: 23-Sep-2022
    • (2022)Detecting Significant Differences Between Information Retrieval Systems via Generalized Linear ModelsProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557286(446-456)Online publication date: 17-Oct-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media