skip to main content
10.1145/2566486.2567998acmotherconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

Was this review helpful to you?: it depends! context and voting patterns in online content

Published: 07 April 2014 Publication History

Abstract

When a website hosting user-generated content asks users a straightforward question - "Was this content helpful?" with one "Yes" and one "No" button as the two possible answers - one might expect to get a straightforward answer. In this paper, we explore how users respond to this question and find that their responses are not quite straightforward after all. Using data from Amazon product reviews, we present evidence that users do not make absolute, independent voting decisions based on individual review quality alone. Rather, whether users vote at all, as well as the polarity of their vote for any given review, depends on the context in which they view it - reviews receive a larger overall number of votes when they are 'misranked', and the polarity of votes becomes more positive/negative when the review is ranked lower/higher than it deserves. We distill these empirical findings into a new probabilistic model of rating behavior that includes the dependence of rating decisions on context. Understanding and formally modeling voting behavior is crucial for designing learning mechanisms and algorithms for review ranking, and we conjecture that many of our findings also apply to user behavior in other online content-rating settings.

References

[1]
E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne. Finding high-quality content in social media. In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, pages 183--194, New York, NY, USA, 2008. ACM.
[2]
G. Askalidis and G. Stoddard. A theoretical analysis of crowdsourced content curation. In The 3rd Workshop on Social Computing and User Generated Content, 2013.
[3]
C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, and L. Lee. How opinions are received by online communities: A case study on amazon.com helpfulness votes. In Proceedings of the 18th international conference on World wide web, WWW '09, pages 141--150, New York, NY, USA, 2009. ACM.
[4]
A. Ghosh and P. McAfee. Incentivizing high-quality user-generated content. In Proceedings of the 20th international conference on World wide web, WWW '11, pages 137--146, New York, NY, USA, 2011. ACM.
[5]
F. M. Harper, D. Raban, S. Rafaeli, and J. A. Konstan. Predictors of answer quality in online Q&A sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 865--874, New York, NY, USA, 2008. ACM.
[6]
N. Jindal and B. Liu. Review spam detection. In Proceedings of the 16th international conference on World Wide Web, WWW '07, pages 1189--1190, New York, NY, USA, 2007. ACM.
[7]
T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2), April 2007.
[8]
S.-M. Kim, P. Pantel, T. Chklovski, and M. Pennacchiotti. Automatically assessing review helpfulness. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP '06, pages 423--430, Stroudsburg, PA, USA, 2006. Association for Computational Linguistics.
[9]
J. Liu, Y. Cao, C. Y. Lin, Y. Huang, and M. Zhou. Low-Quality Product Review Detection in Opinion Summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 334--342, 2007.
[10]
Y. Liu, X. Huang, A. An, and X. Yu. Modeling and predicting the helpfulness of online reviews. In Data Mining, 2008. ICDM '08. Eighth IEEE International Conference on, pages 443--452, 2008.
[11]
Y. Lu, P. Tsaparas, A. Ntoulas, and L. Polanyi. Exploiting social context for review quality prediction. In Proceedings of the 19th international conference on World wide web, WWW '10, pages 691--700, New York, NY, USA, 2010. ACM.
[12]
L. Muchnik, S. Aral, and S. J. Taylor. Social in uence bias: A randomized experiment. Science, 341(6146):647--651, 2013.
[13]
S. M. Mudambi and D. Schuff. What makes a helpful online review? A study of customer reviews on amazon.com. MIS Q., 34(1):185--200, Mar. 2010.
[14]
M. Nakayama and Y. Wan. An exploratory study: "Blind-testing" consumers how they rate helpfulness of online reviews. In Conf-IRM, 2012.
[15]
S. Sen, F. M. Harper, A. LaPitz, and J. Riedl. The quest for quality tags. In Proceedings of the 2007 international ACM conference on Supporting group work, GROUP '07, pages 361--370, New York, NY, USA, 2007. ACM.
[16]
S. Siersdorfer, S. Chelaru, W. Nejdl, and J. San Pedro. How useful are your comments?: Analyzing and predicting youtube comments and comment ratings. In Proceedings of the 19th international conference on World wide web, WWW '10, pages 891--900, New York, NY, USA, 2010. ACM.
[17]
F. Wu and B. Huberman. How public opinion forms. In C. Papadimitriou and S. Zhang, editors, Internet and Network Economics, volume 5385 of Lecture Notes in Computer Science, pages 334--341. Springer Berlin Heidelberg, 2008.
[18]
Z. Zhang and B. Varadarajan. Utility scoring of product reviews. In Proceedings of the 15th ACM international conference on Information and knowledge management, CIKM '06, pages 51--57, New York, NY, USA, 2006. ACM.

Cited By

View all
  • (2023)The Influence of Herd Mentality on Rating Bias and Popularity Bias: A Bi-Process Debiasing Recommendation Model Based on Matrix FactorizationBehavioral Sciences10.3390/bs1301006313:1(63)Online publication date: 10-Jan-2023
  • (2023)D-HRSPTelematics and Informatics10.1016/j.tele.2023.10200182:COnline publication date: 1-Aug-2023
  • (2021)Predicting eWOM’s Influence on Purchase Intention Based on Helpfulness, Credibility, Information Quality and ProfessionalismSustainability10.3390/su1313748613:13(7486)Online publication date: 5-Jul-2021
  • Show More Cited By

Index Terms

  1. Was this review helpful to you?: it depends! context and voting patterns in online content

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        WWW '14: Proceedings of the 23rd international conference on World wide web
        April 2014
        926 pages
        ISBN:9781450327442
        DOI:10.1145/2566486

        Sponsors

        • IW3C2: International World Wide Web Conference Committee

        In-Cooperation

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 April 2014

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. human computation
        2. online reviews
        3. ranking
        4. ratings
        5. user feedback
        6. user-generated content

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        WWW '14
        Sponsor:
        • IW3C2

        Acceptance Rates

        WWW '14 Paper Acceptance Rate 84 of 645 submissions, 13%;
        Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)34
        • Downloads (Last 6 weeks)1
        Reflects downloads up to 05 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2023)The Influence of Herd Mentality on Rating Bias and Popularity Bias: A Bi-Process Debiasing Recommendation Model Based on Matrix FactorizationBehavioral Sciences10.3390/bs1301006313:1(63)Online publication date: 10-Jan-2023
        • (2023)D-HRSPTelematics and Informatics10.1016/j.tele.2023.10200182:COnline publication date: 1-Aug-2023
        • (2021)Predicting eWOM’s Influence on Purchase Intention Based on Helpfulness, Credibility, Information Quality and ProfessionalismSustainability10.3390/su1313748613:13(7486)Online publication date: 5-Jul-2021
        • (2021)The Effect of Online Q&As and Product Reviews on Product Performance Metrics: Amazon.com as a Case StudyJournal of Information & Knowledge Management10.1142/S021964922150005220:01(2150005)Online publication date: 12-Mar-2021
        • (2019)Bandit Learning with Biased Human FeedbackProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3306127.3331838(1324-1332)Online publication date: 8-May-2019
        • (2019)Feature selection for helpfulness prediction of online product reviews: An empirical studyPLOS ONE10.1371/journal.pone.022690214:12(e0226902)Online publication date: 23-Dec-2019
        • (2019)Quantifying Voter Biases in Online PlatformsProceedings of the ACM on Human-Computer Interaction10.1145/33592223:CSCW(1-27)Online publication date: 7-Nov-2019
        • (2018)From Helpfulness Prediction to Helpful Review Retrieval for Online Product ReviewsProceedings of the 9th International Symposium on Information and Communication Technology10.1145/3287921.3287931(38-45)Online publication date: 6-Dec-2018
        • (2018)Helpfulness Prediction of Online Product ReviewsProceedings of the ACM Symposium on Document Engineering 201810.1145/3209280.3229105(1-4)Online publication date: 28-Aug-2018
        • (2018)An Experimental Study of Cryptocurrency Market DynamicsProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174179(1-13)Online publication date: 21-Apr-2018
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media