research-article

Choice of Implicit Signal Matters: Accounting for User Aspirations in Podcast Recommendations

Authors:

Praveen Chandar,

Ghazal Fazelnia,

Catherine M. Edwards,

Benjamin Carterette,

Mounia LalmasAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 2433 - 2441

https://doi.org/10.1145/3485447.3512115

Published: 25 April 2022 Publication History

Abstract

Recommender systems are modulating what billions of people are exposed to on a daily basis. Typically, these systems are optimized for user engagement signals such as clicks, streams, likes, or a weighted combination of such sets. Despite the pervasiveness of this practice, little research has been done to explore the downstream impacts of optimization choice on users, creators and the ecosystem they are offered in. We used a platform that caters recommendations to millions of people and show in practice what you optimize for can have a large impact on the content users are exposed to, as well as what they end up consuming.

In this work, we use podcast recommendations with two engagement signals: Subscription vs. Plays to show that the choice of user engagement matters. We deployed recommendation models optimized for each signal in production and observed that consumption outcomes substantially defer depending on the target used. Upon further investigation, we observed that users’ patterns of podcast engagement depend on the type of podcast, and each podcast can cater to specific user goals & needs. Optimizing for streams can bias the recommendations towards certain podcast types, undermine users’ aspirational interests and put some show categories at disadvantage. Finally, using calibration we demonstrate that informed balanced recommendations can help address this issue and thereby satisfy diverse user interests.

References

[1]

Taleb Alashkar, Songyao Jiang, Shuyang Wang, and Yun Fu. 2017. Examples-rules guided deep neural network for makeup recommendation. In Thirty-First AAAI Conference on Artificial Intelligence.

Digital Library

[2]

Xavier Amatriain and Justin Basilico. 2016. Past, present, and future of recommender systems: An industry perspective. In Proceedings of the 10th ACM Conference on Recommender Systems. 211–214.

Digital Library

[3]

Monica Chadha, Alex Avila, and Homero Gil de Zúñiga. 2012. Listening in: Building a profile of podcast users and analyzing their political participation. Journal of Information Technology & Politics 9, 4 (2012), 388–401.

[4]

Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019. Top-k off-policy correction for a REINFORCE recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 456–464.

Digital Library

[5]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. ACM, 7–10.

Digital Library

[6]

Michael D Ekstrand and Martijn C Willemsen. 2016. Behaviorism is not enough: better recommendations through listening to users. In Proceedings of the 10th ACM Conference on Recommender Systems. 221–224.

Digital Library

[7]

Charles Elkan and Keith Noto. 2008. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 213–220.

Digital Library

[8]

Simon Fietze. 2009. Podcast in higher education: Students usage behaviour. Same places, different spaces. Proceedings ascilite Auckland 2009 (2009), 314–318.

[9]

Pamela L Gay, Rebecca Bemrose-Fetter, Georgia Bracey, and Fraser Cain. 2007. Astronomy Cast: Evaluation of a podcast audience’s content needs and listening habits. Communicating Astronomy with the Public 1, 1 (2007), 24–29.

[10]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining. Ieee, 263–272.

Digital Library

[11]

Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Morgane Lustman, Vince Gatto, Paul Covington, 2019. Reinforcement learning for slate-based recommender systems: A tractable decomposition and practical methodology. arXiv preprint arXiv:1905.12767(2019).

[12]

Dietmar Jannach, Lukas Lerche, and Markus Zanker. 2018. Recommending based on implicit feedback. In Social Information Access. Springer, 510–569.

[13]

Gawesh Jawaheer, Peter Weller, and Patty Kostkova. 2014. Modeling user preferences in recommender systems: A classification framework for explicit and implicit user feedback. ACM Transactions on Interactive Intelligent Systems (TiiS) 4, 2(2014), 1–26.

Digital Library

[14]

Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007(2014).

[15]

Bart P Knijnenburg, Saadhika Sivakumar, and Daricia Wilkinson. 2016. Recommender systems for self-actualization. In Proceedings of the 10th ACM Conference on Recommender Systems. 11–14.

Digital Library

[16]

Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled Experiments on the Web: Survey and Practical Guide. Data Min. Knowl. Discov. 18, 1 (2009), 140–181.

Digital Library

[17]

Steven McClung and Kristine Johnson. 2010. Examining the motives of podcast users. Journal of Radio & Audio Media 17, 1 (2010), 82–95.

[18]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.

[19]

Katherine L Milkman, Todd Rogers, and Max H Bazerman. 2008. Harnessing our inner angels and demons: What we have learned about want/should conflicts and how that knowledge can help us reduce short-sighted decision making. Perspectives on Psychological Science 3, 4 (2008), 324–338.

[20]

Zahra Nazari, Christophe Charbuillet, Johan Pages, Martin Laurent, Denis Charrier, Briana Vecchione, and Ben Carterette. 2020. Recommending Podcasts for Cold-Start Users Based on Music Listening and Taste. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1041–1050.

Digital Library

[21]

Douglas W Oard, Jinmook Kim, 1998. Implicit feedback for recommender systems. In Proceedings of the AAAI workshop on recommender systems, Vol. 83. WoUongong.

[22]

Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. 2008. One-class collaborative filtering. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, 502–511.

Digital Library

[23]

Daniel Read, George Loewenstein, and Shobana Kalyanaraman. 1999. Mixing virtue and vice: Combining the immediacy effect and the diversification heuristic. Journal of Behavioral Decision Making 12, 4 (1999), 257–273.

[24]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618(2012).

[25]

Harald Steck. 2018. Calibrated recommendations. In Proceedings of the 12th ACM conference on recommender systems. 154–162.

Digital Library

[26]

Choon Hui Teo, Houssam Nassif, Daniel Hill, Sriram Srinivasan, Mitchell Goodman, Vijai Mohan, and S.V.N. Vishwanathan. 2016. Adaptive, Personalized Diversity for Visual Discovery. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA, 35–38. https://doi.org/10.1145/2959100.2959171

Digital Library

[27]

Sabina Tomkins, Steven Isley, Ben London, and Lise Getoor. 2018. Sustainability at Scale: Towards Bridging the Intention-Behavior Gap with Sustainable Recommendations(RecSys ’18). Association for Computing Machinery, New York, NY, USA, 214–218. https://doi.org/10.1145/3240323.3240411

Digital Library

[28]

Saúl Vargas and Pablo Castells. 2011. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. In Proceedings of the Fifth ACM Conference on Recommender Systems (Chicago, Illinois, USA) (RecSys ’11). Association for Computing Machinery, New York, NY, USA, 109–116. https://doi.org/10.1145/2043932.2043955

Digital Library

[29]

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems. 279–287.

Digital Library

[30]

Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. 2019. How intention informed recommendations modulate choices: A field study of spoken word content. In The World Wide Web Conference. 2169–2180.

Digital Library

[31]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR) 52, 1 (2019), 1–38.

Digital Library

[32]

Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019. Recommending What Video to Watch next: A Multitask Ranking System. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhagen, Denmark) (RecSys ’19). Association for Computing Machinery, New York, NY, USA, 43–51. https://doi.org/10.1145/3298689.3346997

Digital Library

[33]

Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019. Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2810–2818.

Digital Library

Cited By

Park JLee CCho EOh U(2024)Enhancing the Podcast Browsing Experience through Topic Segmentation and Visualization with Generative AIProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3656324(117-128)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3639701.3656324
Lin KMansoury MEskandanian FSabouri MMobasher BBoratto LGena CMarras MGermanakos PPopescus E(2024)Beyond Static Calibration: The Impact of User Preference Dynamics on Calibrated RecommendationAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664869(86-91)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664869
Loepp B(2023)Multi-list interfaces for recommender systems: survey and future directionsFrontiers in Big Data10.3389/fdata.2023.12397056Online publication date: 10-Aug-2023
https://doi.org/10.3389/fdata.2023.1239705
Show More Cited By

Index Terms

Choice of Implicit Signal Matters: Accounting for User Aspirations in Podcast Recommendations
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
  2. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

How Serendipity Improves User Satisfaction with Recommendations? A Large-Scale User Evaluation
WWW '19: The World Wide Web Conference

Recommendation serendipity is being increasingly recognized as being equally important as the other beyond-accuracy objectives (such as novelty and diversity), in eliminating the “filter bubble” phenomenon of the traditional recommender systems. However,...
Naïve filterbots for robust cold-start recommendations
KDD '06: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining

The goal of a recommender system is to suggest items of interest to a user based on historical behavior of a community of users. Given detailed enough history, item-based collaborative filtering (CF) often performs as well or better than almost any ...
User Personality and User Satisfaction with Recommender Systems

In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
434
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Park JLee CCho EOh U(2024)Enhancing the Podcast Browsing Experience through Topic Segmentation and Visualization with Generative AIProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3656324(117-128)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3639701.3656324
Lin KMansoury MEskandanian FSabouri MMobasher BBoratto LGena CMarras MGermanakos PPopescus E(2024)Beyond Static Calibration: The Impact of User Preference Dynamics on Calibrated RecommendationAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664869(86-91)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664869
Loepp B(2023)Multi-list interfaces for recommender systems: survey and future directionsFrontiers in Big Data10.3389/fdata.2023.12397056Online publication date: 10-Aug-2023
https://doi.org/10.3389/fdata.2023.1239705
Sanner SBalog KRadlinski FWedin BDixon L(2023)Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based PreferencesProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608845(890-896)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608845
Liang YPonnada ALamere PDaskalova N(2023)Enabling Goal-Focused Exploration of Podcasts in Interactive Recommender SystemsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584032(142-155)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584032
da Silva DDurão F(2023)Introducing a framework and a decision protocol to calibrated recommender systemsApplied Intelligence10.1007/s10489-023-04681-753:19(22044-22072)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1007/s10489-023-04681-7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten