skip to main content
10.1145/3290605.3300535acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

How do People Sort by Ratings?

Published: 02 May 2019 Publication History

Abstract

Sorting items by user rating is a fundamental interaction pattern of the modern Web, used to rank products (Amazon), posts (Reddit), businesses (Yelp), movies (YouTube), and more. To implement this pattern, designers must take in a distribution of ratings for each item and define a sensible total ordering over them. This is a challenging problem, since each distribution is drawn from a distinct sample population, rendering the most straightforward method of sorting --- comparing averages --- unreliable when the samples are small or of different sizes. Several statistical orderings for binary ratings have been proposed in the literature (e.g., based on the Wilson score, or Laplace smoothing), each attempting to account for the uncertainty introduced by sampling. In this paper, we study this uncertainty through the lens of human perception, and ask "How do people sort by ratings?" In an online study, we collected 48,000 item-ranking pairs from 4,000 crowd workers along with 4,800 rationales, and analyzed the results to understand how users make decisions when comparing rated items. Our results shed light on the cognitive models users employ to choose between rating distributions, which sorts of comparisons are most contentious, and how the presentation of rating information affects users' preferences.

References

[1]
Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is Seeing Believing?: How Recommender System Interfaces Affect Users' Opinions. In Proc. SIGCHI. 585--592.
[2]
F. Maxwell Harper, Xin Li, Yan Chen, and Joseph A. Konstan. 2005. An Economic Model of User Rating in an Online Recommender System. In Proc. UM. 307--316.
[3]
Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004. Evaluating Collaborative Filtering Recommender Systems. ACM Trans. Inf. Syst. 22 (2004), 5--53.
[4]
Will Hill, Larry Stead, Mark Rosenstein, and George Furnas. 1995. Recommending and Evaluating Choices in a Virtual Community of Use. In Proc. CHI. 194--201.
[5]
Christopher K. Hsee, George F. Loewenstein, Sally Blount, and Max H. Bazerman. 1999. Preference reversals between joint and separate evaluation of options: A review and theoretical analysis. Psychological Bulletin 125, 5 (1999), 576--590. CHI 2019, May 4--9, 2019, Glasgow, Scotland Uk J. Talton et al.
[6]
Nan Hu, Jie Zhang, and Paul A. Pavlou. 2009. Overcoming the J-shaped Distribution of Product Reviews. CACM 52 (2009), 144--147.
[7]
Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux, New York.
[8]
Daniel Kahneman and Amos Tversky. 1979. Prospect Theory: An Analysis of Decision under Risk. Econometrica 47, 2 (1979), 263--291.
[9]
Daniel Kluver, Tien T. Nguyen, Michael Ekstrand, Shilad Sen, and John Riedl. 2012. How Many Bits Per Rating?. In Proc. RecSys. 99--106.
[10]
Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval 3, 3 (2009), 225--331.
[11]
Nathan McAlone. 2017. The exec who replaced Netflix's 5-star rating system with 'thumbs up, thumbs down' explains why. http://www.businessinsider.com/ why-netflix-replaced-its-5-star-rating-system-2017--4
[12]
Evan Miller. 2009. How Not To Sort By Average Rating. http://www. evanmiller.org/how-not-to-sort-by-average-rating.html
[13]
Evan Miller. 2012. Bayesian Average Ratings. http://www.evanmiller. org/bayesian-average-ratings.html
[14]
Evan Miller. 2014. Ranking Items With Star Ratings. http://www. evanmiller.org/how-not-to-sort-by-average-rating.html
[15]
Michael P. O'Mahony, Neil J. Hurley, and Guénolé C.M. Silvestre. 2006. Detecting Noise in Recommender System Databases. In Proc. IUI. 109-- 115.
[16]
Will Qiu, Palo Parigi, and Bruno Abrahao. 2018. More Stars or More Reviews?. In Proc. CHI. 153:1--153:11.
[17]
Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, and John Riedl. 2002. Getting to Know You: Learning New User Preferences in Recommender Systems. In Proc. IUI. 127--134.
[18]
Alan Said and Alejandro Bellogín. 2018. Coherence and Inconsistencies in Rating Behavior: Estimating the Magic Barrier of Recommender Systems. UMUAI 28 (2018), 97--125.
[19]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based Collaborative Filtering Recommendation Algorithms. In Proc. WWW. 285--295.
[20]
Aaron Schumacher. 2014. How To Sort By Average Rating. https://planspacedotorg.wordpress.com/2014/08/17/ how-to-sort-by-average-rating/
[21]
Stefan Siersdorfer, Sergiu Chelaru, Wolfgang Nejdl, and Jose San Pedro. 2010. How Useful Are Your Comments?: Analyzing and Predicting Youtube Comments and Comment Ratings. In Proc. WWW. 891--900.
[22]
E. Isaac Sparling and Shilad Sen. 2011. Rating: How Difficult is It?. In Proc. RecSys. 149--156.
[23]
Jacob Thebault-Spieker, Daniel Kluver, Maximilian A. Klein, Aaron Halfaker, Brent Hecht, Loren Terveen, and Joseph A. Konstan. 2017. Simulation Experiments on (the Absence of) Ratings Bias in Reputation Systems. In Proc. CSCW. 101:1--101:25.
[24]
Amos Tversky and Daniel Kahneman. 1985. The Framing of Decisions and the Psychology of Choice. Springer US, Boston, MA, 25--41.
[25]
Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference. J. Amer. Statist. Assoc. 22, 158 (1927), 209--212.
[26]
Timothy Wilson and Jonathan Schooler. 1991. Thinking Too Much: Introspection Can Reduce the Quality of Preferences and Decisions. Journal of personality and social psychology 60 (03 1991), 181--92.
[27]
Dell Zhang, Robert Mao, Haitao Li, and Joanne Mao. 2011. How to Count Thumb-Ups and Thumb-Downs: User-Rating Based Ranking of Items from an Axiomatic Perspective. In Proc ICTIR. 238--249.

Cited By

View all
  • (2023)Understanding of Customer Decision-Making Behaviors Depending on Online ReviewsApplied Sciences10.3390/app1306394913:6(3949)Online publication date: 20-Mar-2023
  • (2023)Implementation of Wilson Score on Personal Information Distribution for Privacy-Focused Contact ManagementProceedings of the International Conference on Educational Management and Technology (ICEMT 2022)10.2991/978-2-494069-95-4_55(464-478)Online publication date: 10-Feb-2023
  • (2021)3 Stars on Yelp, 4 Stars on Google MapsProceedings of the ACM on Human-Computer Interaction10.1145/34329534:CSCW3(1-25)Online publication date: 5-Jan-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
May 2019
9077 pages
ISBN:9781450359702
DOI:10.1145/3290605
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ranking
  2. uncertainty
  3. user ratings

Qualifiers

  • Research-article

Conference

CHI '19
Sponsor:

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)3
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Understanding of Customer Decision-Making Behaviors Depending on Online ReviewsApplied Sciences10.3390/app1306394913:6(3949)Online publication date: 20-Mar-2023
  • (2023)Implementation of Wilson Score on Personal Information Distribution for Privacy-Focused Contact ManagementProceedings of the International Conference on Educational Management and Technology (ICEMT 2022)10.2991/978-2-494069-95-4_55(464-478)Online publication date: 10-Feb-2023
  • (2021)3 Stars on Yelp, 4 Stars on Google MapsProceedings of the ACM on Human-Computer Interaction10.1145/34329534:CSCW3(1-25)Online publication date: 5-Jan-2021
  • (2021)Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the FieldProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445734(1-16)Online publication date: 6-May-2021
  • (2021)Imma Sort by Two or More Attributes With Interpretable Monotonic Multi-Attribute SortingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.304348727:4(2369-2384)Online publication date: 1-Apr-2021
  • (2020)Utilizing Response Time to Find In-between Ratings within Likes and DislikesExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3383049(1-7)Online publication date: 25-Apr-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media