skip to main content
10.1145/2365952.2365974acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

How many bits per rating?

Published: 09 September 2012 Publication History

Abstract

Most recommender systems assume user ratings accurately represent user preferences. However, prior research shows that user ratings are imperfect and noisy. Moreover, this noise limits the measurable predictive power of any recommender system. We propose an information theoretic framework for quantifying the preference information contained in ratings and predictions. We computationally explore the properties of our model and apply our framework to estimate the efficiency of different rating scales for real world datasets. We then estimate how the amount of information predictions give to users is related to the scale ratings are collected on. Our findings suggest a tradeoff in rating scale granularity: while previous research indicates that coarse scales (such as thumbs up / thumbs down) take less time, we find that ratings with these scales provide less predictive value to users. We introduce a new measure, preference bits per second, to quantitatively reconcile this tradeoff.

References

[1]
X. Amatriain, J. Pujol, and N. Oliver. I like it... i like it not: Evaluating user ratings noise in recommender systems. In UMAP 2009, pages 247--258. Springer (2009), 2009.
[2]
X. Amatriain, J. M. Pujol, N. Tintarev, and N. Oliver. Rate it again: increasing recommendation accuracy by user re-rating. In RecSys 09. ACM, 2009.
[3]
D. Cosley, S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. Is seeing believing?: How recommender system interfaces affect users' opinions. In CHI 03. ACM, 2003.
[4]
T. M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, 2006.
[5]
M. Ekstrand, M. Ludwig, J. Konstan, and J. Riedl. Rethinking the recommender research ecosystem: Reproducibility, openness, and LensKit. In RecSys 11, pages 133--140. ACM, 2011.
[6]
M. D. Ekstrand, J. T. Riedl, and J. A. Konstan. Collaborative filtering recommender system. Foundations and Trends in Human-Computer Interaction, 4(2):81--173, 2010.
[7]
B. Fischhoff. Value elicitation: is there anything in there? American Psychologist, 46(8):835, 1991.
[8]
W. R. Garner. Rating scales, discriminability, and information transmission. The Psychological Review, 67(6):343--352, 1960.
[9]
K. Goldberg, T. Roeder, D. Gupta, and C. Perkins. Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval, 4(2):133--151, 2001.
[10]
F. Harper, X. Li, Y. Chen, and J. Konstan. An economic model of user rating in an online recommender system. User Modeling, pages 149--149, 2005.
[11]
J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst., 22(1):5--53, Jan. 2004.
[12]
W. Hill, L. Stead, M. Rosenstein, and G. Furnas. Recommending and evaluating choices in a virtual community of use. In CHI 95, pages 194--201, 1995.
[13]
G. Miller. Note on the bias of information estimates. Information theory in psychology: Problems and methods, 2:95--100, 1955.
[14]
M. P. O'Mahony, N. J. Hurley, N. Kushmerick, and G. Silvestre. Collaborative recommendation: A robustness analysis. ACM Transactions on Internet Technology, 4:344--377, 2004.
[15]
M. P. O'Mahony, N. J. Hurley, and G. C. Silvestre. Detecting noise in recommender system databases. In IUI 06. ACM, 2006.
[16]
L. Paninski. Estimation of entropy and mutual information. Neural Computation, 15(6):1191--1253, 2003.
[17]
B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In WWW 2001, pages 285--295. ACM, 2001.
[18]
S. Sen, J. Vig, and J. Riedl. Tagommenders: Connecting users to items through tags. In WWW 09, pages 671--680. ACM, 2009.
[19]
G. Shani and A. Gunawardana. Evaluating recommendation systems. Recommender Systems Handbook, pages 257--297, 2011.
[20]
C. Shannon. A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1):3--55, 2001.
[21]
E. I. Sparling and S. Sen. Rating: How difficult is it? In RecSys 11. ACM, 2011.
[22]
Youtube. New video page launches for all users @http://youtube-global.blogspot.com/2010/03/new-video-page-launches-for-all-users.html, Mar. 2010.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '12: Proceedings of the sixth ACM conference on Recommender systems
September 2012
376 pages
ISBN:9781450312707
DOI:10.1145/2365952
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2012

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. evaluation
  2. information theory
  3. metrics
  4. ratings
  5. recommender systems

Qualifiers

  • Research-article

Conference

RecSys '12
Sponsor:
RecSys '12: Sixth ACM Conference on Recommender Systems
September 9 - 13, 2012
Dublin, Ireland

Acceptance Rates

RecSys '12 Paper Acceptance Rate 24 of 119 submissions, 20%;
Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)1
Reflects downloads up to 02 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Natural noise management in collaborative recommender systems over time-related informationThe Journal of Supercomputing10.1007/s11227-024-06267-7Online publication date: 8-Jul-2024
  • (2024)Interactive Recommendation SystemsHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_54-1(1-29)Online publication date: 11-Feb-2024
  • (2023)Handling PreferencesGroup Recommender Systems10.1007/978-3-031-44943-7_5(95-107)Online publication date: 23-Sep-2023
  • (2021)Human‐centered recommender systemsAI Magazine10.1609/aimag.v42i3.1814242:3(31-42)Online publication date: 1-Sep-2021
  • (2021)Effective Strategies for Crowd-Powered Cognitive Reappraisal Systems: A Field Deployment of the Flip*Doubt Web Application for Mental HealthProceedings of the ACM on Human-Computer Interaction10.1145/34795615:CSCW2(1-37)Online publication date: 18-Oct-2021
  • (2021)Critique on Natural Noise in Recommender SystemsACM Transactions on Knowledge Discovery from Data10.1145/344778015:5(1-30)Online publication date: 29-May-2021
  • (2021)Modelling user reactions expressed through graphical widgets in intelligent interactive systemsBehaviour & Information Technology10.1080/0144929X.2021.193145041:11(2438-2483)Online publication date: 1-Jun-2021
  • (2020)“Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for RecommendationProceedings of the 14th ACM Conference on Recommender Systems10.1145/3383313.3412267(398-407)Online publication date: 22-Sep-2020
  • (2019)Optimal Number of Choices in Rating ContextsBig Data and Cognitive Computing10.3390/bdcc30300483:3(48)Online publication date: 27-Aug-2019
  • (2019)When actions speak louder than clicksProceedings of the 13th ACM Conference on Recommender Systems10.1145/3298689.3347044(287-295)Online publication date: 10-Sep-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media