skip to main content
10.1145/3640457.3688144acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

The Fault in Our Recommendations: On the Perils of Optimizing the Measurable

Published: 08 October 2024 Publication History

Abstract

Recommendation systems are widespread, and through customized recommendations, promise to match users with options they will like. To that end, data on engagement is collected and used. Most recommendation systems are ranking-based, where they rank and recommend items based on their predicted engagement. However, the engagement signals are often only a crude proxy for user utility, as data on the latter is rarely collected or available. This paper explores the following question: By optimizing for measurable proxies, are recommendation systems at risk of significantly under-delivering on user utility? If that is indeed the case, how can one improve utility which is seldom measured? To study these questions, we introduce a model of repeated user consumption in which, at each interaction, users select between an outside option and the best option from a recommendation set. Our model accounts for user heterogeneity, with the majority preferring “popular” content, and a minority favoring “niche” content. The system initially lacks knowledge of individual user preferences but can learn these preferences through observations of users’ choices over time. Our theoretical and numerical analysis demonstrate that optimizing for engagement signals can lead to significant utility losses. Instead, we propose a utility-aware policy that initially recommends a mix of popular and niche content. We show that such a policy substantially improves utility despite not measuring it. As the platform becomes more forward-looking, our utility-aware policy achieves the best of both worlds: near-optimal user utility and near-optimal engagement simultaneously. Our study elucidates an important feature of recommendation systems; given the ability to suggest multiple items, one can perform significant exploration without incurring significant reductions in short term engagement. By recommending high-risk, high-reward items alongside popular items, systems can enhance discovery of high utility items without significantly affecting engagement.

Supplemental Material

MP4 File - The Fault in Our Recommendations: On the Perils of Optimizing the Measurable
A 2 minute summary of the paper "The Fault in Our Recommendations: On the Perils of Optimizing the Measurable" by Omar Besbes, Yash Kanoria and Akshit Kumar.

References

[1]
Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. In Proceedings of the eleventh ACM conference on recommender systems. 42–46.
[2]
Gediminas Adomavicius and Alexander Tuzhilin. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering 17, 6 (2005), 734–749.
[3]
Abdul Basit Ahanger, Syed Wajid Aalam, Muzafar Rasool Bhat, and Assif Assad. 2022. Popularity bias in recommender systems-a review. In International Conference on Emerging Technologies in Computer Engineering. Springer, 431–444.
[4]
Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, and Mounia Lalmas. 2020. Algorithmic effects on the diversity of consumption on spotify. In Proceedings of the web conference 2020. 2155–2165.
[5]
Simon P Anderson, Andre De Palma, and Jacques-Francois Thisse. 1992. Discrete choice theory of product differentiation. MIT press.
[6]
Barry C Arnold. 2008. Pareto and generalized Pareto distributions. In Modeling income distributions and Lorenz curves. Springer, 119–145.
[7]
Omar Besbes, Yonatan Gur, and Assaf Zeevi. 2016. Optimization in Online Content Recommendation services: Beyond Click-Through-Rates. Manufacturing & Service Operations Management 18, 1 (2016), 15–33.
[8]
Òscar Celma and Pedro Cano. 2008. From hits to niches? or how popular artists can bias music recommendation and discovery. In Proceedings of the 2nd KDD workshop on large-scale recommender systems and the netflix prize competition. 1–8.
[9]
Minmin Chen, Yuyan Wang, Can Xu, Ya Le, Mohit Sharma, Lee Richardson, Su-Lin Wu, and Ed Chi. 2021. Values of user exploration in recommender systems. In Proceedings of the 15th ACM Conference on Recommender Systems. 85–95.
[10]
Tom Cunningham, Sana Pandey, Leif Sigerson, Jonathan Stray, Jeff Allen, Bonnie Barrilleaux, Ravi Iyer, Smitha Milli, Mohit Kothari, and Behnam Rezaei. 2024. What We Know About Using Non-Engagement Signals in Content Ranking. arXiv preprint arXiv:2402.06831 (2024).
[11]
Michael D Ekstrand and Martijn C Willemsen. 2016. Behaviorism is not enough: better recommendations through listening to users. In Proceedings of the 10th ACM conference on recommender systems. 221–224.
[12]
Jonathan L Herlocker, Joseph A Konstan, Loren G Terveen, and John T Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 5–53.
[13]
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Manish Raghavan. 2023. The inversion problem: Why algorithms should infer mental state and not just predict behavior. Perspectives on Psychological Science (2023), 17456916231212138.
[14]
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2023. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management Science (2023).
[15]
Anastasiia Klimashevskaia, Dietmar Jannach, Mehdi Elahi, and Christoph Trattner. 2023. A Survey on Popularity Bias in Recommender Systems. arXiv preprint arXiv:2308.01118 (2023).
[16]
Tor Lattimore and Csaba Szepesvári. 2020. Bandit algorithms. Cambridge University Press.
[17]
Thomas M McDonald, Lucas Maystre, Mounia Lalmas, Daniel Russo, and Kamil Ciosek. 2023. Impatient bandits: Optimizing recommendations for the long-term without delay. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1687–1697.
[18]
Smitha Milli, Luca Belli, and Moritz Hardt. 2021. From optimizing engagement to measuring value. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 714–722.
[19]
Smitha Milli, Emma Pierson, and Nikhil Garg. 2023. Choosing the right weights: Balancing value, strategy, and noise in recommender systems. arXiv preprint arXiv:2305.17428 (2023).
[20]
Carey K Morewedge, Sendhil Mullainathan, Haaya F Naushan, Cass R Sunstein, Jon Kleinberg, Manish Raghavan, and Jens O Ludwig. 2023. Human bias in algorithm design. Nature Human Behaviour 7, 11 (2023), 1822–1824.
[21]
Yoon-Joo Park and Alexander Tuzhilin. 2008. The long tail of recommender systems and how to leverage it. In Proceedings of the 2008 ACM conference on Recommender systems. 11–18.
[22]
Kenny Peng, Manish Raghavan, Emma Pierson, Jon Kleinberg, and Nikhil Garg. 2024. Reconciling the accuracy-diversity trade-off in recommendations. In Proceedings of the ACM on Web Conference 2024. 1318–1329.
[23]
João Sá, Vanessa Queiroz Marinho, Ana Rita Magalhães, Tiago Lacerda, and Diogo Goncalves. 2022. Diversity vs relevance: A practical multi-objective study in luxury fashion recommendations. In Proceedings of the 45th International ACM SIGIR Conference on research and development in information retrieval. 2405–2409.
[24]
Harald Steck. 2018. Calibrated recommendations. In Proceedings of the 12th ACM conference on recommender systems. 154–162.
[25]
Jonathan Stray, Ivan Vendrov, Jeremy Nixon, Steven Adler, and Dylan Hadfield-Menell. 2021. What are you optimizing for? aligning recommender systems with human values. arXiv preprint arXiv:2107.10939 (2021).

Index Terms

  1. The Fault in Our Recommendations: On the Perils of Optimizing the Measurable

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      RecSys '24: Proceedings of the 18th ACM Conference on Recommender Systems
      October 2024
      1438 pages
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 08 October 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Discrete Choice Models
      2. Engagement-Utility Trade-off
      3. Recommendation Systems

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      Acceptance Rates

      Overall Acceptance Rate 254 of 1,295 submissions, 20%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 305
        Total Downloads
      • Downloads (Last 12 months)305
      • Downloads (Last 6 weeks)21
      Reflects downloads up to 18 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media