Update Delivery Mechanisms for Prospective Information Needs: A Reproducibility Study
Pages 308 - 312
Abstract
Real-time summarization systems monitor continuous streams of documents with the goal of delivering relevant, novel, and timely updates to users. These updates can either be sent to users' mobile devices as push notifications or be silently deposited in an inbox to be consumed - the important difference is whether the user is interrupted by the delivery. Previously, a two-year study examining user attention under these different mechanisms revealed interesting findings about users' information consumption behavior, but the conclusions were marred by a few methodological shortcomings. We present a reproducibility study that follows the same design as the original evaluation, but corrects its flaws. We find that most conclusions from the original study are confirmed, although there are some surprising differences as well. Overall, the magnitude of the observed effects are not as strong as in the original study.
References
[1]
James Allan. 2002. Topic Detection and Tracking: Event-Based Information Organization .Kluwer Academic Publishers, Dordrecht, The Netherlands.
[2]
Jaime Arguello, Matt Crane, Fernando Diaz, Jimmy Lin, and Andrew Trotman. 2015. Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR) . SIGIR Forum, Vol. 49, 2 (2015), 107--116.
[3]
Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, and Ze Zhong Wu. 2019. Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019). In Proceedings of the Open-Source IR Replicability Challenge at SIGIR 2019: CEUR Workshop Proceedings Vol-2409 . Paris, France, 1--7.
[4]
Nicola Ferro. 2017. Reproducibility Challenges in Information Retrieval Evaluation. Journal of Data and Information Quality, Vol. 8, 2 (2017), Article 8.
[5]
Nicola Ferro, Norbert Fuhr, Kalervo J"arvelin, Noriko Kando, Matthias Lippold, and Justin Zobel. 2016. Increasing Reproducibility in IR: Findings from the Dagstuhl Seminar on “Reproducibility of Data-Oriented Experiments in e-Science”. SIGIR Forum, Vol. 50, 1 (2016), 68--82.
[6]
Nicola Ferro and Diane Kelly. 2018. SIGIR Initiative to Implement ACM Artifact Review and Badging. SIGIR Forum, Vol. 52, 1 (2018), 4--10.
[7]
Frank Hopfgartner, Allan Hanbury, Henning Müller, Ivan Eggel, Krisztian Balog, Torben Brodt, Gordon V. Cormack, Jimmy Lin, Jayashree Kalpathy-Cramer, Noriko Kando, Makoto P. Kato, Anastasia Krithara, Tim Gollub, Martin Potthast, Evelyne Viegas, and Simon Mercer. 2018. Evaluation-as-a-Service for the Computational Sciences: Overview and Outlook. Journal of Data and Information Quality, Vol. 10, 4 (2018), Article 15.
[8]
David D. Lewis. 1995. The TREC-4 Filtering Track. In Proceedings of the Fourth Text REtrieval Conference (TREC-4). Gaithersburg, Maryland, 165--180.
[9]
Jimmy Lin, Matt Crane, Andrew Trotman, Jamie Callan, Ishan Chattopadhyaya, John Foley, Grant Ingersoll, Craig Macdonald, and Sebastiano Vigna. 2016a. Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016) . Padua, Italy, 408--420.
[10]
Jimmy Lin, Salman Mohammed, Royal Sequiera, and Luchen Tan. 2018. Update Delivery Mechanisms for Prospective Information Needs: An Analysis of Attention in Mobile Users. In Proceedings of the 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018). Ann Arbor, Michigan, 785--794.
[11]
Jimmy Lin, Salman Mohammed, Royal Sequiera, Luchen Tan, Nimesh Ghelani, Mustafa Abualsaud, Richard McCreadie, Dmitrijs Milajevs, and Ellen Voorhees. 2017. Overview of the TREC 2017 Real-Time Summarization Track. In Proceedings of the Twenty-Sixth Text REtrieval Conference (TREC 2017) . Gaithersburg, Maryland.
[12]
Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen Voorhees, and Fernando Diaz. 2016b. Overview of the TREC 2016 Real-Time Summarization Track. In Proceedings of the Twenty-Fifth Text REtrieval Conference (TREC 2016) . Gaithersburg, Maryland.
[13]
Abhinav Mehrotra, Robert Hendley, and Mirco Musolesi. 2016a. PrefMiner: Mining User's Preferences for Intelligent Mobile Notification Management. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2016) . Heidelberg, Germany, 1223--1234.
[14]
Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016b. My Phone and Me: Understanding People's Receptivity to Mobile Notifications. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016). Santa Clara, California, 1021--1032.
[15]
Anne Schuth, Krisztian Balog, and Liadh Kelly. 2015. Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015. In Proceedings of CLEF 2015 . Toulouse, France, 484--496.
[16]
Royal Sequiera, Luchen Tan, and Jimmy Lin. 2018. Overview of the TREC 2018 Real-Time Summarization Track. In Proceedings of the Twenty-Seventh Text REtrieval Conference (TREC 2018) . Gaithersburg, Maryland.
[17]
Luchen Tan, Gaurav Baruah, and Jimmy Lin. 2017. On the Reusability of “Living Labs” Test Collections: A Case Study of Real-Time Summarization. In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017). Tokyo, Japan, 793--796.
Index Terms
- Update Delivery Mechanisms for Prospective Information Needs: A Reproducibility Study
Recommendations
Update Delivery Mechanisms for Prospective Information Needs: An Analysis of Attention in Mobile Users
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information RetrievalReal-time summarization systems that monitor document streams to identify relevant content have a few options for delivering system updates to users. In a mobile context, systems could send push notifications to users' mobile devices, hoping to grab ...
Beyond Interruptibility: Predicting Opportune Moments to Engage Mobile Phone Users
Many of today's mobile products and services engage their users proactively via push notifications. However, such notifications are not always delivered at the right moment, therefore not meeting products' and users' expectations. To address this ...
Enabling smartphone push notifications: the effect of a framed opt-in request
The use of push notifications is one of the most relevant strategies to proactively communicate with a user from within various apps. Europe has adopted an opt-in requirement wherein users have to explicitly express that they agree to receive push ...
Comments
Information & Contributors
Information
Published In
March 2020
596 pages
ISBN:9781450368926
DOI:10.1145/3343413
- General Chairs:
- Heather O'Brien,
- Luanne Freund,
- Program Chairs:
- Ioannis Arapakis,
- Orland Hoeber,
- Irene Lopatovska
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 March 2020
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
CHIIR '20: Conference on Human Information Interaction and Retrieval
March 14 - 18, 2020
Vancouver BC, Canada
Acceptance Rates
Overall Acceptance Rate 55 of 163 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 104Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in