skip to main content
10.1145/3240323.3240384acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
short-paper

Streamingrec: a framework for benchmarking stream-based news recommenders

Published: 27 September 2018 Publication History

Abstract

News is one of the earliest application domains of recommender systems, and recommending items from a virtually endless stream of news is still a relevant problem today. News recommendation is different from other application domains in a variety of ways, e.g., because new items constantly become available for recommendation. To be effective, news recommenders therefore have to continuously consider the latest items in the incoming stream of news in their recommendation models. However, today's public software libraries for algorithm benchmarking mostly do not consider these particularities of the domain. As a result, authors often rely on proprietary protocols, which hampers the comparability of the obtained results. In this paper, we present StreamingRec as a framework for evaluating streaming-based news recommenders in a replicable way. The open-source framework implements a replay-based evaluation protocol that allows algorithms to update the underlying models in real-time when new events are recorded and new articles are available for recommendation. Furthermore, a variety of baseline algorithms for session-based recommendation are part of StreamingRec. For these, we also report a number of performance results for two datasets, which confirm the importance of immediate model updates.

References

[1]
Shlomo Berkovsky, Jill Freyne, and Gregory Smith. 2012. Personalized Network Updates: Increasing Social Interactions and Contributions in Social Networks. In Proceedings of the 20th International Conference on User Modeling, Adaptation, and Personalization (UMAP '12). 1--13.
[2]
Daniel Billsus, Michael J. Pazzani, and James Chen. 2000. A Learning Agent for Wireless News Access. In Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI '00). 33--36.
[3]
Pedro G. Campos, Fernando Díez, and Iván Cantador. 2014. Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction 24, 1 (2014), 67--119.
[4]
Iván Cantador and Pablo Castells. 2009. Semantic contextualisation in a news recommender system. In Proceedings of the 2009 Workshop on Context-Aware Recommender Systems (CARS '09).
[5]
Badrish Chandramouli, Justin J. Levandoski, Ahmed Eldawy, and Mohamed F. Mokbel. 2011. StreamRec: A Real-time Recommender System. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (SIGMOD '11). 1243--1246.
[6]
Wei Chu and Seung-Taek Park. 2009. Personalized Recommendation on Dynamic Content Using Predictive Bilinear Models. In Proceedings of the 18th International Conference on World Wide Web (WWW '09). 691--700.
[7]
Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google News Personalization: Scalable Online Collaborative Filtering. In Proceedings of the 16th International Conference on World Wide Web (WWW '07). 271--280.
[8]
Doychin Doychev, Aonghus Lawlor, Rachael Rafter, and Barry Smyth. 2014. An Analysis of Recommender Algorithms for Online News. In Working Notes for the 2014 Conference and Labs of the Evaluation Forum (CLEF '14). 825--836.
[9]
Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John T. Riedl. 2011. Rethinking the Recommender Research Ecosystem: Reproducibility, Openness, and LensKit. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). 133--140.
[10]
Blaz Fortuna, Carolina Fortuna, and Dunja Mladenic. 2010. Real-Time News Recommender System. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD '10). 583--586.
[11]
Erzsébet Frigó, Róbert Pálovics, Domokos Kelen, Levente Kocsis, and András A. Benczúr. 2017. Alpenglow: Open Source Recommender Framework with Time-aware Learning and Evaluation. In Proceedings of the Poster Track of the 11th ACM Conference on Recommender Systems (RecSys '17).
[12]
Erzsébet Frigó, Róbert Pálovics, Domokos Kelen, Levente Kocsis, and András A. Benczúr. 2017. Online Ranking Prediction in Non-stationary Environments. In Proceedings of the 1st Workshop on Temporal Reasoning in Recommender Systems co-located with RecSys '17 (RecTemp '17). 28--34.
[13]
Zeno Gantner, Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. MyMediaLite: A Free Recommender System Library. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). 305--308.
[14]
Qi Gao, Fabian Abel, Geert-Jan Houben, and Ke Tao. 2011. Interweaving Trend and User Modeling for Personalized News Recommendation. In Proceedings of the 2011 International Conferences on Web Intelligence and Intelligent Agent Technology (WI-IAT '11). 100--103.
[15]
Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and Online Evaluation of News Recommender Systems at Swissinfo.Ch. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys '14). 169--176.
[16]
Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec: A Java Library for Recommender Systems. In Posters, Demos, Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modelling, Adaptation and Personalization (UMAP '15).
[17]
Ido Guy, Naama Zwerdling, Inbal Ronen, David Carmel, and Erel Uziel. 2010. Social Media Recommendation Based on People and Tags. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '10). 194--201.
[18]
Jiawei Han, Jian Pei, Yiwen Yin, and Runying Mao. 2004. Mining Frequent Patterns without Candidate Generation: A Frequent-Pattern Tree Approach. Data Mining and Knowledge Discovery 8, 1 (2004), 53--87.
[19]
Balázs Hidasi and Alexandros Karatzoglou. 2017. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. CoRR (2017). arXiv:1706.03847 {cs.LG} http://arxiv.org/abs/1706.03847
[20]
Frank Hopfgartner, Torben Brodt, Jonas Seiler, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Roberto Turrin, and András Serény. 2016. Benchmarking News Recommendations: The CLEF NewsREEL Use Case. SIGIR Forum 49, 2 (2016), 129--136.
[21]
Frank Hopfgartner, Benjamin Kille, Tobias Heintz, and Roberto Turrin. 2015. Real-time Recommendation of Streamed Data. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). 361--362.
[22]
Yanxiang Huang, Bin Cui, Wenyu Zhang, Jie Jiang, and Ying Xu. 2015. TencentRec: Real-time Stream Recommendation in Practice. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD '15). 227--238.
[23]
Dietmar Jannach, Lukas Lerche, and Michael Jugovac. 2015. Adaptation and Evaluation of Recommendations for Short-term Shopping Goals. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). 211--218.
[24]
Dietmar Jannach and Malte Ludewig. 2017. When Recurrent Neural Networks Meet the Neighborhood for Session-Based Recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys '17). 306--310.
[25]
Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender Systems - Beyond Matrix Completion. Commun. ACM 59, 11 (2016), 94--102.
[26]
Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. 2018. News recommender systems---Survey and roads ahead. Information Processing & Management (2018). In press.
[27]
Mohammad Khoshnesh in and W. Nick Street. 2010. Incremental Collaborative Filtering via Evolutionary Co-clustering. In Proceedings of the Fourth ACM Conference on Recommender Systems (RecSys '10). 325--328.
[28]
Benjamin Kille, Frank Hopfgartner, Torben Brodt, and Tobias Heintz. 2013. The Plista Dataset. In Proceedings of the 2013 International News Recommender Systems Workshop and Challenge (NRS '13). 16--23.
[29]
Evan Kirshenbaum, George Forman, and Michael Dugan. 2012. A Live Comparison of Methods for Personalized Article Recommendation at Forbes.com. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD '12). 51--66.
[30]
Takuya Kitazawa. 2017. FluRS: A Python Library for Online Item Recommendation. (2017). Retrieved April 30, 2018 from https://takuti.me/note/flurs/
[31]
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A Contextual-bandit Approach to Personalized News Article Recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW '10). 661--670.
[32]
Lei Li and Tao Li. 2013. News recommendation via hypergraph learning: encapsulation of user behavior and news content. In Proceedings of the 6th International Conference on Web Search and Data Mining (WSDM '13). 305--314.
[33]
Lei Li, Li Zheng, and Tao Li. 2011. LOGO: A Long-Short User Interest Integration in Personalized News Recommendation. In Proceedings of the 5th Conference on Recommender Systems (RecSys '11). 317--320.
[34]
Lei Li, Li Zheng, Fan Yang, and Tao Li. 2014. Modeling and broadening temporal user interest in personalized news recommendation. Expert Systems with Applications 41, 7 (2014), 3168 -- 3177.
[35]
Chen Lin, Runquan Xie, Xinjun Guan, Lei Li, and Tao Li. 2014. Personalized news recommendation via implicit social experts. Information Sciences 254 (2014), 1--18.
[36]
Malte Ludewig and Dietmar Jannach. 2018. Evaluation of Session-Based Recommenation Algorithms. (2018). arXiv:1803.09587 {cs.IR} https://arxiv.org/abs/1803.09587
[37]
Cornelius A. Ludmann. 2017. Recommending News Articles in the CLEF News Recommendation Evaluation Lab with the Data Stream Management System Odysseus. In Working Notes for the 2017 Conference and Labs of the Evaluation Forum (CLEF '17).
[38]
Owen Phelan, Kevin McCarthy, and Barry Smyth. 2009. Using Twitter to Recommend Real-time Topical News. In Proceedings of the Third ACM Conference on Recommender Systems (RecSys '09). 385--388.
[39]
Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-Aware Recommender Systems. Comput. Surveys (2018). arXiv:1802.08452 {cs.IR} https://arxiv.org/abs/1802.08452 forthcoming.
[40]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09). 452--461.
[41]
Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW '94). 175--186.
[42]
Alan Said and Alejandro Bellogín. 2014. Rival: A Toolkit to Foster Reproducibility in Recommender System Evaluation. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys '14). 371--372.
[43]
Mario Scriminaci, Andreas Lommatzsch, Benjamin Kille, Frank Hopfgartner, Martha Larson, Davide Malagoli, András Serény, and Till Plumbaum. 2016. Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. In Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems (RecSys '16).
[44]
Bichen Shi, Georgiana Ifrim, and Neil Hurley. 2016. Learning-to-Rank for Real-Time High-Precision Hashtag Recommendation for Streaming News. In Proceedings of the 25th International Conference on World Wide Web (WWW '16). 1191--1202.
[45]
Yong Kiam Tan, Xinxing Xu, and Yong Liu. 2016. Improved Recurrent Neural Networks for Session-based Recommendations. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS '16). 17--22.
[46]
Hao Wen, Liping Fang, and Ling Guan. 2012. A hybrid approach for personalized recommendation of news on the Web. Expert Systems with Applications 39, 5 (2012), 5806--5814.

Cited By

View all
  • (2024)Bridging the GapARO-THE SCIENTIFIC JOURNAL OF KOYA UNIVERSITY10.14500/aro.1151912:1(100-107)Online publication date: 3-Apr-2024
  • (2024)Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00289(3767-3781)Online publication date: 13-May-2024
  • (2024)Session-based News Recommendation Using Cohesive Patterns2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825982(440-447)Online publication date: 15-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems
September 2018
600 pages
ISBN:9781450359016
DOI:10.1145/3240323
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. benchmarking
  2. evaluation
  3. news recommendation

Qualifiers

  • Short-paper

Conference

RecSys '18
Sponsor:
RecSys '18: Twelfth ACM Conference on Recommender Systems
October 2, 2018
British Columbia, Vancouver, Canada

Acceptance Rates

RecSys '18 Paper Acceptance Rate 32 of 181 submissions, 18%;
Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)19
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Bridging the GapARO-THE SCIENTIFIC JOURNAL OF KOYA UNIVERSITY10.14500/aro.1151912:1(100-107)Online publication date: 3-Apr-2024
  • (2024)Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00289(3767-3781)Online publication date: 13-May-2024
  • (2024)Session-based News Recommendation Using Cohesive Patterns2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825982(440-447)Online publication date: 15-Dec-2024
  • (2024)SessionPrint: Accelerating kNN via Locality-Sensitive Hashing for Session-Based News RecommendationExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-71736-9_10(159-165)Online publication date: 14-Sep-2024
  • (2023)Dually Enhanced Delayed Feedback Modeling for Streaming Conversion Rate PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614856(390-399)Online publication date: 21-Oct-2023
  • (2023)When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591785(942-952)Online publication date: 19-Jul-2023
  • (2023)Comparison of Real-Time and Batch Job RecommendationsIEEE Access10.1109/ACCESS.2023.324935611(20553-20559)Online publication date: 2023
  • (2023)Leveraging Sequential Episode Mining for Session-Based News RecommendationWeb Information Systems Engineering – WISE 202310.1007/978-981-99-7254-8_46(594-608)Online publication date: 21-Oct-2023
  • (2022)Streaming Session-Based Recommendation: When Graph Neural Networks meet the NeighborhoodProceedings of the 16th ACM Conference on Recommender Systems10.1145/3523227.3548485(420-426)Online publication date: 12-Sep-2022
  • (2022)Positive, Negative and NeutralProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532040(1185-1195)Online publication date: 6-Jul-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media