skip to main content
10.1145/1645953.1646293acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
poster

Evaluation of methods for relative comparison of retrieval systems based on clickthroughs

Published: 02 November 2009 Publication History

Abstract

The Cranfield evaluation method has some disadvantages, including its high cost in labor and inadequacy for evaluating interactive retrieval techniques. As a very promising alternative, automatic comparison of retrieval systems based on observed clicking behavior of users has recently been studied. Several methods have been proposed, but there has so far been no systematic way to assess which strategy is better, making it difficult to choose a good method for real applications. In this paper, we propose a general way to evaluate these relative comparison methods with two measures: utility to users(UtU) and effectiveness of differentiation(EoD). We evaluate two state of the art methods by systematically simulating different retrieval scenarios. Inspired by the weakness of these methods revealed through our evaluation, we further propose a novel method by considering the positions of clicked documents. Experiment results show that our new method performs better than the existing methods.

References

[1]
Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais. Here or there. In ECIR, volume 4956 of Lecture Notes in Computer Science, pages 16--27. Springer, 2008.
[2]
T. Joachims. Unbiased evaluation of retrieval quality using clickthrough data. In SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval, 2002.
[3]
T. Joachims, L. Granka, Bing Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2), April 2007.
[4]
F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In Conference on Information and Knowledge Management (CIKM), 2008.

Cited By

View all
  • (2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
  • (2023)Stat-Weight: Improving the Estimator of Interleaved Methods Outcomes with Statistical Hypothesis TestingAdvances in Information Retrieval10.1007/978-3-031-28241-6_2(20-34)Online publication date: 16-Mar-2023
  • (2022)Debiased Balanced Interleaving at Amazon SearchProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557123(2913-2922)Online publication date: 17-Oct-2022
  • Show More Cited By

Index Terms

  1. Evaluation of methods for relative comparison of retrieval systems based on clickthroughs

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management
    November 2009
    2162 pages
    ISBN:9781605585123
    DOI:10.1145/1645953
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 November 2009

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. evaluation
    2. implicit feedback
    3. information retrieval

    Qualifiers

    • Poster

    Conference

    CIKM '09
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
    • (2023)Stat-Weight: Improving the Estimator of Interleaved Methods Outcomes with Statistical Hypothesis TestingAdvances in Information Retrieval10.1007/978-3-031-28241-6_2(20-34)Online publication date: 16-Mar-2023
    • (2022)Debiased Balanced Interleaving at Amazon SearchProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557123(2913-2922)Online publication date: 17-Oct-2022
    • (2020)Counterfactual Online Learning to RankAdvances in Information Retrieval10.1007/978-3-030-45439-5_28(415-430)Online publication date: 8-Apr-2020
    • (2019)Position Bias Estimation for Unbiased Learning-to-Rank in eCommerce SearchString Processing and Information Retrieval10.1007/978-3-030-32686-9_4(47-64)Online publication date: 3-Oct-2019
    • (2019)Optimizing Ranking Models in an Online SettingAdvances in Information Retrieval10.1007/978-3-030-15712-8_25(382-396)Online publication date: 7-Apr-2019
    • (2018)Differentiable Unbiased Online Learning to RankProceedings of the 27th ACM International Conference on Information and Knowledge Management10.1145/3269206.3271686(1293-1302)Online publication date: 17-Oct-2018
    • (2017)Sensitive and Scalable Online Evaluation with Theoretical GuaranteesProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3132895(77-86)Online publication date: 6-Nov-2017
    • (2016)Constructing Reliable Gradient Exploration for Online Learning to RankProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983774(1643-1652)Online publication date: 24-Oct-2016
    • (2016)Interleaved Evaluation for Retrospective Summarization and Prospective Notification on Document StreamsProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2911494(175-184)Online publication date: 7-Jul-2016
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media