skip to main content
10.1145/3269206.3269242acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
short-paper

Empirical Evidence for Search Effectiveness Models

Published: 17 October 2018 Publication History

Abstract

Given a SERP in response to a user-originated query, Moffat et al. (CIKM 2013; TOIS 2017) suggest that C(i), the conditional continuation probability of the user examining the (i+1)st element presented in the SERP, given that they are known to have examined the ith one, is positively correlated with both i and with the user's initial estimate of the volume of answer pages they are looking for, and negatively correlated with the extent to which suitable answer pages have been identified in the SERP at positions 1 through i. Here we first describe a methodology for specifying how C(i) should be defined in practical (as against ideal) settings, and then evaluate the applicability of the approach using three large search interaction logs from two different sources.

References

[1]
A. Ashkan and C. L. A. Clarke. On the informativeness of cascade and intent-aware effectiveness measures. In Proc. WWW, pages 407--416, 2011.
[2]
L. Azzopardi, P. Thomas, and N. Craswell. Measuring the utility of search engine result pages: An information foraging measure. In Proc. SIGIR, pages 605--614, 2018.
[3]
B. Carterette. System effectiveness, user models, and user utility: A conceptual framework for investigation. In Proc. SIGIR, pages 903--912, 2011.
[4]
O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proc. CIKM, pages 621--630, 2009.
[5]
J. Jiang and J. Allan. Adaptive persistence for search effectiveness measures. In Proc. CIKM, pages 747--756, 2017.
[6]
A. Moffat and A. F. Wicaksono. Users, adaptivity, and bad abandonment. In Proc. SIGIR, pages 897--900, 2018.
[7]
A. Moffat, P. Thomas, and F. Scholer. Users versus models: What observation tells us about effectiveness metrics. In Proc. CIKM, pages 659--668, 2013.
[8]
A. Moffat, P. Bailey, F. Scholer, and P. Thomas. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Trans. Inf. Sys., 35 (3): 24:1--24:38, 2017.
[9]
P. Thomas, F. Scholer, and A. Moffat. What users do: The eyes have it. In Proc. Asia Info. Retri. Soc. Conf., pages 416--427, 2013.

Cited By

View all
  • (2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
  • (2024)AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance AssessmentProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698420(54-63)Online publication date: 8-Dec-2024
  • (2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management
October 2018
2362 pages
ISBN:9781450360142
DOI:10.1145/3269206
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptive metric
  2. average precision
  3. evaluation
  4. user model

Qualifiers

  • Short-paper

Funding Sources

Conference

CIKM '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,029 of 4,238 submissions, 24%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
  • (2024)AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance AssessmentProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698420(54-63)Online publication date: 8-Dec-2024
  • (2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
  • (2023)A Reference-Dependent Model for Web Search EvaluationProceedings of the ACM Web Conference 202310.1145/3543507.3583551(3396-3405)Online publication date: 30-Apr-2023
  • (2022)A Flexible Framework for Offline Effectiveness MetricsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531924(578-587)Online publication date: 6-Jul-2022
  • (2022)Efficient query processing techniques for next-page retrievalInformation Retrieval Journal10.1007/s10791-021-09402-7Online publication date: 18-Jan-2022
  • (2021)ERR is not C/W/L: Exploring the Relationship Between Expected Reciprocal Rank and Other MetricsProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472239(231-237)Online publication date: 11-Jul-2021
  • (2021)Incorporating Query Reformulating Behavior into Web Search EvaluationProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482438(171-180)Online publication date: 26-Oct-2021
  • (2021)Modeling search and session effectivenessInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10260158:4Online publication date: 1-Jul-2021
  • (2020)Models Versus SatisfactionProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401162(379-388)Online publication date: 25-Jul-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media