skip to main content
10.1145/2766462.2767821acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper

Evaluating Retrieval Models through Histogram Analysis

Published:09 August 2015Publication History

ABSTRACT

We present a novel approach for efficiently evaluating the performance of retrieval models and introduce two evaluation metrics: Distributional Overlap (DO), which compares the clustering of scores of relevant and non-relevant documents, and Histogram Slope Analysis (HSA), which examines the log of the empirical distributions of relevant and non-relevant documents. Unlike rank evaluation metrics such as mean average precision (MAP) and normalized discounted cumulative gain (NDCG), DO and HSA only require calculating model scores of queries and a fixed sample of relevant and non-relevant documents rather than scoring the entire collection, even implicitly by means of an inverted index. In experimental meta-evaluations, we find that HSA achieves high correlation with MAP and NDCG on a monolingual and a cross-language document similarity task; on four ad-hoc web retrieval tasks; and on an analysis of ten TREC tasks from the past ten years. In addition, when evaluating latent Dirichlet allocation (LDA) models on document similarity tasks, HSA achieves better correlation with MAP and NCDG than perplexity, an intrinsic metric widely used with topic models.

References

  1. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. JMLR, 3: 993--1022, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. W. Croft. A model of cluster searching based on classification. Information Systems, 5 (3): 189--195, 1980.Google ScholarGoogle ScholarCross RefCross Ref
  3. N. Jardine and C. J. van Rijsbergen. The use of hierarchical clustering in information retrieval. Information Storage and Retrieval, 7: 217--240, 1971.Google ScholarGoogle ScholarCross RefCross Ref
  4. K. Krstovski and D. A. Smith. Online polylingual topic models for fast document translation detection. In WMT'11, pages 252--261, 2013.Google ScholarGoogle Scholar
  5. D. Mimno, H. Wallach, J. Naradowsky, D. A. Smith, and A. McCallum. Polylingual topic models. In EMNLP'09, pages 880--889, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. F. Raiber and O. Kurland. The correlation between cluster hypothesis tests and the effectiveness of cluster-based retrieval. In SIGIR '14, pages 1155--1158, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In CIKM '07, pages 623--632, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. C. van Rijsbergen. Automatic Information Structuring and Retrieval. PhD thesis, University of Cambridge, 1972.Google ScholarGoogle Scholar
  9. E. M. Voorhees. The cluster hypothesis revisited. In SIGIR '85, pages 188--196, 1985. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. X. Xue and W. B. Croft. Transforming patents into prior-art queries. In SIGIR '09, pages 808--809, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Evaluating Retrieval Models through Histogram Analysis

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval
      August 2015
      1198 pages
      ISBN:9781450336215
      DOI:10.1145/2766462

      Copyright © 2015 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 9 August 2015

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • short-paper

      Acceptance Rates

      SIGIR '15 Paper Acceptance Rate70of351submissions,20%Overall Acceptance Rate792of3,983submissions,20%
    • Article Metrics

      • Downloads (Last 12 months)2
      • Downloads (Last 6 weeks)0

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader