short-paper

Public Access

The Pareto Frontier of Utility Models as a Framework for Evaluating Push Notification Systems

Authors:

Gaurav Baruah,

Jimmy LinAuthors Info & Claims

ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

Pages 253 - 256

https://doi.org/10.1145/3121050.3121089

Published: 01 October 2017 Publication History

PDF eReader

Abstract

We propose a utility-based framework for the evaluation of push notification systems that monitor document streams for users' topics of interest. Our starting point is that users derive either positive utility (i.e., "gain") or negative utility (i.e., "pain") from consuming system updates. By separately keeping track of these quantities, we can measure system effectiveness in a gain vs. pain tradeoff space. The Pareto Frontier of evaluated systems represents the state of the art: for each system on the frontier, no other system can offer more gain without more pain. Our framework has several advantages: it unifies three previous TREC evaluations, subsumes existing metrics, and provides more insightful analyses. Furthermore, our approach can easily accommodate more refined user models and is extensible to different information-seeking modalities.

References

[1]

James Allan. 2002. Topic Detection and Tracking: Event-Based Information Organization. Kluwer Academic Publishers, Dordrecht, The Netherlands.

Digital Library

Google Scholar

[2]

Javed Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Richard McCreadie, Fernando Diaz, and Tetsuya Sakai 2014. TREC 2014 Temporal Summarization Track Overview TREC.

Google Scholar

[3]

Javed Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Fernando Diaz, and Tetsuya Sakai. 2013. TREC 2013 Temporal Summarization. In TREC.

Google Scholar

[4]

Leif Azzopardi. 2016. Simulation of Interaction: A Tutorial on Modelling and Simulating User Interaction and Search Behaviour. In SIGIR. 1227--1230.

Digital Library

Google Scholar

[5]

Leif Azzopardi, Kalervo Jarvelin, Jaap Kamps, and Mark D. Smucker 2011. Report on the SIGIR 2010 Workshop on the Simulation of Interaction. SIGIR Forum, Vol. 44, 2 (2011), 35--47.

Digital Library

Google Scholar

[6]

Gaurav Baruah, Mark D. Smucker, and Charles L. A. Clarke. 2015. Evaluating Streams of Evolving News Events. In SIGIR. 675--684.

Digital Library

Google Scholar

[7]

Charles L. A. Clarke and Mark D. Smucker 2014. Time Well Spent IIiX '14. 205--214.

Digital Library

Google Scholar

[8]

Qi Guo, Fernando Diaz, and Elad Yom-Tov 2013. Updating Users about Time Critical Events. In ECIR. 483--494.

Digital Library

Google Scholar

[9]

Heikki Keskustalo, Kalervo Jarvelin, Ari Pirkola, and Jaana Kekalainen 2008. Intuition-Supporting Visualization of User's Performance Based on Explicit Negative Higher-Order Relevance. In SIGIR. 675--682.

Digital Library

Google Scholar

[10]

David D. Lewis. 1995. The TREC-4 Filtering Track. In TREC. 165--180.

Google Scholar

[11]

Jimmy Lin, Miles Efron, Yulu Wang, and Garrick Sherman. 2015. Overview of the TREC-2015 Microblog Track. TREC.

Google Scholar

[12]

Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen Voorhees, and Fernando Diaz. 2016. Overview of the TREC 2016 Real-Time Summarization Track TREC.

Google Scholar

[13]

David Maxwell, Leif Azzopardi, Kalervo Jarvelin, and Heikki Keskustalo 2015. Searching and Stopping: An Analysis of Stopping Rules and Strategies CIKM. 313--322.

Digital Library

Google Scholar

[14]

Alistair Moffat and Justin Zobel 2008. Rank-Biased Precision for Measurement of Retrieval Effectiveness. ACM TOIS, Vol. 27, 1, Article 2 (Dec. 2008), 27 pages.

Digital Library

Google Scholar

Cited By

View all

Sombattheera C(2022)News Feed: A Multiagent-Based Push Notification SystemMulti-disciplinary Trends in Artificial Intelligence10.1007/978-3-031-20992-5_11(120-132)Online publication date: 10-Nov-2022
https://doi.org/10.1007/978-3-031-20992-5_11
Fang HKamps JKanoulas Ede Rijke MYilmaz E(2018)Report on the 2017 ACM SIGIR International Conference Theory of Information Retrieval (ICTIR?17)ACM SIGIR Forum10.1145/3190580.319059151:3(78-87)Online publication date: 22-Feb-2018
https://dl.acm.org/doi/10.1145/3190580.3190591

Index Terms

The Pareto Frontier of Utility Models as a Framework for Evaluating Push Notification Systems
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Retrieval tasks and goals
      1. Document filtering

Recommendations

A Study of Realtime Summarization Metrics
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for ...
Approximating Pareto frontier using a hybrid line search approach

The aggregation of objectives in multiple criteria programming is one of the simplest and widely used approach. But it is well known that this technique sometimes fail in different aspects for determining the Pareto frontier. This paper proposes a new ...
A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management

There is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task ...

Comments

Information & Contributors

Information

Published In

ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

October 2017

348 pages

ISBN:9781450344906

DOI:10.1145/3121050

General Chairs:
Jaap Kamps
University of Amsterdam, The Netherlands
,
Evangelos Kanoulas
University of Amsterdam, The Netherlands
,
Maarten de Rijke
University of Amsterdam, The Netherlands
,
Program Chairs:
Hui Fang
University of Delaware, USA
,
Emine Yilmaz
University College London, UK

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Short Paper

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

ICTIR '17

Sponsor:

SIGIR

ICTIR '17: ACM SIGIR International Conference on the Theory of Information Retrieval

October 1 - 4, 2017

Amsterdam, The Netherlands

Acceptance Rates

ICTIR '17 Paper Acceptance Rate 27 of 54 submissions, 50%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
145
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)7

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sombattheera C(2022)News Feed: A Multiagent-Based Push Notification SystemMulti-disciplinary Trends in Artificial Intelligence10.1007/978-3-031-20992-5_11(120-132)Online publication date: 10-Nov-2022
https://doi.org/10.1007/978-3-031-20992-5_11
Fang HKamps JKanoulas Ede Rijke MYilmaz E(2018)Report on the 2017 ACM SIGIR International Conference Theory of Information Retrieval (ICTIR?17)ACM SIGIR Forum10.1145/3190580.319059151:3(78-87)Online publication date: 22-Feb-2018
https://dl.acm.org/doi/10.1145/3190580.3190591

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A Study of Realtime Summarization Metrics

Approximating Pareto frontier using a hybrid line search approach

A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries