skip to main content
10.1145/3640457.3688183acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
short-paper

It's (not) all about that CTR: A Multi-Stakeholder Perspective on News Recommender Metrics

Published: 08 October 2024 Publication History

Abstract

Recommender systems are increasingly used by news media organizations. Existing literature examines various aspects of news recommender systems (NRS) from a computational, user-centric, or normative perspective. Yet research advocates studying the complexities of real-world applications around NRS. Recently, a multi-stakeholder approach to NRS has been adopted, allowing to understand different stakeholder perspectives on NRS development and evaluation within the news organization. However, little research has been done on the different key performance indicators (KPIs) and metrics considered valuable by different stakeholders. Based on 11 interviews with professionals from two commercial news publishers, this paper demonstrates that stakeholders prioritize distinct KPIs and metrics related to the reach-engagement-conversion-retention funnel. The evaluation of NRS performance is often limited to short-term metrics like CTR, overlooking the multiplicity of stakeholders involved. Our findings reveal how different purposes, KPIs, and metrics are valued from the journalistic, commercial, and tech logic. In doing so, this paper contributes to the multi-stakeholder approach to NRS, advancing our understanding of the real-world complexity of NRS development and evaluation.

References

[1]
Mariella Bastian, Natali Helberger, and Mykola Makhortykh. 2021. Safeguarding the journalistic DNA: Attitudes towards the role of professional values in algorithmic news recommender designs. Digital Journalism 9, 6 (2021), 835–863.
[2]
Christine Bauer, Chandni Bagchi, Olusanmi A Hundogan, and Karin van Es. 2024. Where are the Values? A Systematic Literature Review on News Recommender Systems. ACM Transactions on Recommender Systems (2024).
[3]
Sina Blassnig, Edina Strikovic, Eliza Mitova, Aleksandra Urman, Anikó Hannák, Claes de Vreese, and Frank Esser. 2023. A Balancing Act: How Media Professionals Perceive the Implementation of News Recommender Systems. Digital Journalism (2023), 1–23.
[4]
Balázs Bodó. 2021. Selling news to audiences–a qualitative inquiry into the emerging logics of algorithmic news personalization in European quality news media. In Algorithms, Automation, and News. Routledge, 75–96.
[5]
Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: scalable online collaborative filtering. In Proceedings of the 16th international conference on World Wide Web. 271–280.
[6]
Raul Ferrer-Conill and Edson C Tandoc Jr. 2018. The audience-oriented editor: Making sense of the audience in the newsroom. Digital Journalism 6, 4 (2018), 436–453.
[7]
Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and online evaluation of news recommender systems at swissinfo. ch. In Proceedings of the 8th ACM Conference on Recommender systems. 169–176.
[8]
Brian Halligan and Dharmesh Shah. 2014. Inbound marketing, revised and updated: Attract, engage, and delight customers online. John Wiley & Sons.
[9]
Dietmar Jannach and Gediminas Adomavicius. 2016. Recommendations with a purpose. In Proceedings of the 10th ACM conference on recommender systems. 7–10.
[10]
Dietmar Jannach and Christine Bauer. 2020. Escaping the McNamara fallacy: Towards more impactful recommender systems research. Ai Magazine 41, 4 (2020), 79–95.
[11]
Dietmar Jannach and Michael Jugovac. 2019. Measuring the business value of recommender systems. ACM Transactions on Management Information Systems (TMIS) 10, 4 (2019), 1–23.
[12]
Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. 2018. News recommender systems–Survey and roads ahead. Information Processing & Management 54, 6 (2018), 1203–1227.
[13]
Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010. Personalized news recommendation based on click behavior. In Proceedings of the 15th international conference on Intelligent user interfaces. 31–40.
[14]
Eliza Mitova, Sina Blassnig, Edina Strikovic, Aleksandra Urman, Claes de Vreese, and Frank Esser. 2023. When Worlds Collide: Journalistic, Market, and Tech Logics in the Adoption of News Recommender Systems. Journalism Studies 24, 16 (2023), 1957–1976.
[15]
Eliza Mitova, Sina Blassnig, Edina Strikovic, Aleksandra Urman, Aniko Hannak, Claes H de Vreese, and Frank Esser. 2023. News recommender systems: A programmatic research review. Annals of the International Communication Association 47, 1 (2023), 84–113.
[16]
Lynge Asbjørn Møller. 2022. Recommended for you: how newspapers normalise algorithmic news recommendation to fit their gatekeeping role. Journalism Studies 23, 7 (2022), 800–817.
[17]
Lynge Asbjørn Møller. 2023. Designing Algorithmic Editors: How Newspapers Embed and Encode Journalistic Values into News Recommender Systems. Digital Journalism (2023), 1–19.
[18]
Ragnhild Kristine Olsen, Aske Kammer, and Mona Kristin Solvoll. 2020. Paywalls’ impact on local news websites’ traffic and their civic and business implications. Journalism Studies 21, 2 (2020), 197–216.
[19]
Lawrence A Palinkas, Sarah M Horwitz, Carla A Green, Jennifer P Wisdom, Naihua Duan, and Kimberly Hoagwood. 2015. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and policy in mental health and mental health services research 42 (2015), 533–544.
[20]
Shaina Raza and Chen Ding. 2022. News recommender system: a review of recent progress, challenges, and opportunities. Artificial Intelligence Review (2022), 1–52.
[21]
Jorge Remondes, Paulo Madeira, and Carlos Alves. 2024. Connecting with Consumers Through Effective Personalization and Programmatic Advertising. IGI Global.
[22]
Annelien Smets, Jonathan Hendrickx, and Pieter Ballon. 2022. We’re in this together: a multi-stakeholder approach for news recommenders. Digital Journalism 10, 10 (2022), 1813–1831.
[23]
Iddo Tavory and Stefan Timmermans. 2014. Abductive analysis: Theorizing qualitative research. University of Chicago Press.
[24]
Celina Treuillier, Sylvain Castagnos, Evan Dufraisse, and Armelle Brun. 2022. Being diverse is not enough: Rethinking diversity evaluation to meet challenges of news recommender systems. In Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization. 222–233.
[25]
Sanne Vrijenhoek, Savvina Daniil, Jorden Sandel, and Laura Hollink. 2024. Diversity of What? On the Different Conceptualizations of Diversity in Recommender Systems. arXiv preprint arXiv:2405.02026 (2024).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '24: Proceedings of the 18th ACM Conference on Recommender Systems
October 2024
1438 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Evaluation
  2. Impact of Recommenders
  3. Metrics
  4. Multi-stakeholder
  5. News Recommender Systems

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Conference

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 133
    Total Downloads
  • Downloads (Last 12 months)133
  • Downloads (Last 6 weeks)11
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media