skip to main content
10.1145/3523227.3551479acmotherconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
short-paper

The Effect of Feedback Granularity on Recommender Systems Performance

Published: 13 September 2022 Publication History

Abstract

The main source of knowledge utilized in recommender systems (RS) is users’ feedback. While the usage of implicit feedback (i.e. user’s behavior statistics) is gaining in prominence, the explicit feedback (i.e. user’s ratings) remain an important data source. This is true especially for domains, where evaluation of an object does not require an extensive usage and users are well motivated to do so (e.g., video-on-demand services or library archives).
So far, numerous rating schemes for explicit feedback have been proposed, ranging both in granularity and presentation style. There are several works studying the effect of rating’s scale and presentation on user’s rating behavior, e.g. willingness to provide feedback or various biases in rating behavior. Nonetheless, the effect of ratings granularity on RS performance remain largely under-researched.
In this paper, we studied the combined effect of ratings granularity and supposed probability of feedback existence on various performance statistics of recommender systems. Results indicate that decreasing feedback granularity may lead to changes in RS’s performance w.r.t. nDCG for some recommending algorithms. Nonetheless, in most cases the effect of feedback granularity is surpassed by even a small decrease in feedback’s quantity. Therefore, our results corroborate the policy of many major real-world applications, i.e. preference of simpler rating schemes with the higher chance of feedback reception instead of finer-grained rating scenarios.

Supplementary Material

MP4 File (TheEffectOfFeedbackGranularityOnRSPerformance.mp4)
Brief introduction to ACM RecSys 2022 LBR paper "The Effect of Feedback Granularity on Recommender Systems Performance" by Ladislav Peska

References

[1]
Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling Popularity Bias in Learning-to-Rank Recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems (Como, Italy) (RecSys ’17). Association for Computing Machinery, New York, NY, USA, 42–46. https://doi.org/10.1145/3109859.3109912
[2]
Vito Walter Anelli, Alejandro Bellogín, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021. Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 2405–2414. https://doi.org/10.1145/3404835.3463245
[3]
Federica Cena, Cristina Gena, Pierluigi Grillo, Tsvi Kuflik, Fabiana Vernero, and Alan J. Wecker. 2017. How scales influence user rating behaviour in recommender systems. Behaviour & Information Technology 36, 10 (2017), 985–1004. https://doi.org/10.1080/0144929X.2017.1322145 arXiv:https://doi.org/10.1080/0144929X.2017.1322145
[4]
Shulong Chen and Yuxing Peng. 2018. Matrix factorization for recommendation with explicit and implicit feedback. Knowledge-Based Systems 158 (2018), 109–117. https://doi.org/10.1016/j.knosys.2018.05.040
[5]
F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (dec 2015), 19 pages. https://doi.org/10.1145/2827872
[6]
Yehuda Koren. 2008. Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Las Vegas, Nevada, USA) (KDD ’08). Association for Computing Machinery, New York, NY, USA, 426–434. https://doi.org/10.1145/1401890.1401944
[7]
G. Linden, B. Smith, and J. York. 2003. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing 7, 1 (2003), 76–80. https://doi.org/10.1109/MIC.2003.1167344
[8]
Supriyo Mandal and Abyayananda Maiti. 2020. Explicit feedback meet with implicit feedback in GPMF: a generalized probabilistic matrix factorization model for recommendation. Applied Intelligence 50, 6 (01 Jun 2020), 1955–1978. https://doi.org/10.1007/s10489-020-01643-1
[9]
Carolyn C Preston and Andrew M Colman. 2000. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica 104, 1 (2000), 1–15. https://doi.org/10.1016/S0001-6918(99)00050-5
[10]
Julia Shaftel, Brooke L Nash, and SC Gillmor. 2012. Effects of the number of response categories on rating scales. In Proceedings of the annual conference of the American Educational Research Association. 1–24.
[11]
E. Isaac Sparling and Shilad Sen. 2011. Rating: How Difficult is It?. In Proceedings of the Fifth ACM Conference on Recommender Systems (Chicago, Illinois, USA) (RecSys ’11). Association for Computing Machinery, New York, NY, USA, 149–156. https://doi.org/10.1145/2043932.2043961
[12]
Gábor Takács and Domonkos Tikk. 2012. Alternating Least Squares for Personalized Ranking. In Proceedings of the Sixth ACM Conference on Recommender Systems (Dublin, Ireland) (RecSys ’12). Association for Computing Machinery, New York, NY, USA, 83–90. https://doi.org/10.1145/2365952.2365972
[13]
Paula Cristina Vaz, Ricardo Ribeiro, and David Martins De Matos. 2013. Understanding the Temporal Dynamics of Recommendations across Different Rating Scales. In UMAP Workshops. CEUR-WS.
[14]
Li-Jen Weng. 2004. Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability. Educational and Psychological Measurement 64, 6 (2004), 956–972. https://doi.org/10.1177/0013164404268674 arXiv:https://doi.org/10.1177/0013164404268674
[15]
Baolin Yi, Xiaoxuan Shen, Hai Liu, Zhaoli Zhang, Wei Zhang, Sannyuya Liu, and Naixue Xiong. 2019. Deep Matrix Factorization With Implicit Feedback Embedding for Recommendation System. IEEE Transactions on Industrial Informatics 15, 8 (2019), 4591–4601. https://doi.org/10.1109/TII.2019.2893714
[16]
Tong Zhao, Julian McAuley, and Irwin King. 2015. Improving Latent Factor Models via Personalized Feature Projection for One Class Recommendation. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (Melbourne, Australia) (CIKM ’15). Association for Computing Machinery, New York, NY, USA, 821–830. https://doi.org/10.1145/2806416.2806511

Cited By

View all
  • (2024)Understanding Perceptions of the Reddit Reaction Mechanism in Political SubredditsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681861(261-267)Online publication date: 11-Nov-2024
  • (2024)Exploring the Potential of Generative AI for Augmenting Choice-Based Preference Elicitation in Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664873(114-119)Online publication date: 27-Jun-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems
September 2022
743 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Explicit feedback granularity
  2. Performance evaluation
  3. Recommender systems

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Funding Sources

  • Charles University Grant Agency
  • Czech Science Foundation
  • Charles University

Conference

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)33
  • Downloads (Last 6 weeks)8
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Understanding Perceptions of the Reddit Reaction Mechanism in Political SubredditsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681861(261-267)Online publication date: 11-Nov-2024
  • (2024)Exploring the Potential of Generative AI for Augmenting Choice-Based Preference Elicitation in Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664873(114-119)Online publication date: 27-Jun-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media