skip to main content
10.1145/2911451.2911478acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
abstract

Retrievability: An Independent Evaluation Measure

Published: 07 July 2016 Publication History

Abstract

Information Retrieval systems have traditionally been evaluated in terms of efficiency and performance. These aspects of retrieval systems, whilst very important, do not cover a crucial aspect of the system, the access it provides to the documents of the collection. Retrievability, a document centric evaluation measure, introduced by Azzopardi and Vinay, provides an alternative approach to evaluation [1]. Retrievability is the ease with which a document can be retrieved using a retrieval system. The more queries which retrieve the document, and the higher up the document is returned, the more retrievable it is. It can thus be used to describe how difficult it is to find documents in the collection given a particular configuration of a retrieval system. Unlike typical performance evaluations, performing a retrievability analysis can be done without recourse to relevancy judgements meaning there is no reliance on a test collection. This has major advantages when tuning a retrieval systems parameters as the tuning can be performed on the live collection.

References

[1]
Azzopardi, L., Vinay, V.: Retrievability: An evaluation measure for higher order information access tasks. In: Proc. of the 17th ACM CIKM. pp. 561--570 (2008)
[2]
Wilkie, C., Azzopardi, L.: Relating retrievability, performance and length. In: Proc. of the 36th ACM SIGIR conference. pp. 937--940 (2013)
[3]
Wilkie, C., Azzopardi, L.: Best and fairest: An empirical analysis of retrieval system bias. Advances in Information Retrieval pp. 13--25 (2014)
[4]
Wilkie, C., Azzopardi, L.: Efficiently estimating retrievability bias. In: Advances in Information Retrieval. pp. 720--726 (2014)
[5]
Wilkie, C., Azzopardi, L.: A retrievability analysis: Exploring the relationship between retrieval bias and retrieval performance. In: Proc. of the 23rd ACM CIKM. pp. 81--90 (2014)
[6]
Wilkie, C., Azzopardi, L.: Query length, retrievability bias and performance. In: Proc. of the 24th ACM CIKM. pp. 1787--1790 (2015)
[7]
Wilkie, C., Azzopardi, L.: Retrievability bias: A comparison of inequality measures. Advances in Information Retrieval pp. 209--214 (2015)

Index Terms

  1. Retrievability: An Independent Evaluation Measure

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval
    July 2016
    1296 pages
    ISBN:9781450340694
    DOI:10.1145/2911451
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2016

    Check for updates

    Author Tags

    1. bias
    2. effectiveness
    3. evaluation
    4. retrievability

    Qualifiers

    • Abstract

    Funding Sources

    Conference

    SIGIR '16
    Sponsor:

    Acceptance Rates

    SIGIR '16 Paper Acceptance Rate 62 of 341 submissions, 18%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 157
      Total Downloads
    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media