Skip to main content
Log in

On the inaccuracy of numerical ratings: dealing with biased opinions in social networks

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

In this work, we study the potential problems emanating from using numerical ratings in social networks to rank entities regarding their reputation. In particular, we empirically demonstrate how reputation rankings as collected and managed by current systems are likely to be skewed due to subjectivity problems associated with the use of numerical ratings to encapsulate preferences. With the aim of overcoming these problems, we put forward an approach in which users are asked for their opinions about entities in a comparative fashion. In order to select the most appropriate users to be queried, we take advantage of the social structure derived from the interactions among users and entities following a principle of heterogeneity. Finally, we evaluate the proposed approach in the domain of movie ratings by using real datasets collected from different web sites.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://digg.com/

  2. https://netflix.com/

  3. http://www.movielens.org/

  4. This dataset was used in the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems (See http://ir.ii.uam.es/hetrec2011). It is collected from the MovieLens opinion site, where movies are rated by different users.

  5. As in the MovieLens site, this is an opinion website where movies are rated. However, in this case, movies are also reviewed by professional critics (See http://www.rottentomatoes.com).

  6. http://www.cs.sfu.ca/%7Esja25/personal/datasets/

  7. http://www.flixster.com

  8. http://ir.ii.uam.es/hetrec2011

  9. http://ir.ii.uam.es

  10. http://www.grouplens.org

  11. http://www.movielens.org/

  12. http://www.rottentomatoes.com

References

  • Anderson, M.L., & Magruder, J.R. (2012). Learning from the crowd: Regression discontinuity estimates of the effects of an online review database. Economic Journal, 122 (563), 957–989.

    Article  Google Scholar 

  • Balakrishnan, S., & Chopra, S. (2010). Two of a kind or the ratings game? Adaptive pairwise preferences and latent factor models. Proceedings of the 2010 IEEE international conference on data mining. ICDM ’10, (pp. 725–730). Washington, DC: IEEE Computer Society.

  • Berinsky, A.J. (1999). The two faces of public opinion. American Journal of Political Science, 43 (4), 1209–1230.

    Article  Google Scholar 

  • Brams, S.J., & Fishburn, P.C. (2002). Voting procedures In Arrow, K. J., Sen, A. K., Suzumura, K. (Eds.), Handbook of social choice and welfare (Vol. 1, pp. 173–236). New York: Elsevier.

  • Burke, R. (2000). Knowledge-based Recommender Systems. Encyclopedia of library and information systems Vol. 69.

  • Chen, Li, & Pu, Pearl (2004). Survey of preference elicitation methods. Tech. rep., Swiss Federal Institure of Technology in Lausanne (EPFL).

  • Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J. (2003). Is seeing believing?: how recommender system interfaces affect users’ opinions. Proceedings of the sigchi conference on human factors in computing systems. CHI ’03, (pp. 585–592). New York: ACM.

  • Dalvi, N., Kumar, R., Pang, B. (2013). Para ’normal’ activity: On the distribution of average ratings. Proceedings of 7th international AAAI conference on weblogs and social media, (pp. 110–119).

  • Dellarocas, C. (2000). Mechanisms for coping with unfair ratings and discriminatory behavior in online reputation reporting systems. ICIS ’00: Proceedings of the 21st international conference on information systems, (pp. 520–525). Atlanta: Association for Information Systems.

  • Fowler, J.H., & Christakis, N.A. (2010). Cooperative behavior cascades in human social networks. Proceedings of the National Academy of Sciences, 107 (12), 5334–5338.

    Article  Google Scholar 

  • Fu, F., Hauert, C., Nowak, M.A., Wang, L. (2008). Reputation-based partner choice promotes cooperation in social networks. Physical Review E, 78.

  • Fullam, K.K., & Barber, K.S. (2007). Dynamically learning sources of trust information: experience vs. reputation. Proceedings of AAMAS ’07, (pp. 1055–1062). New York: ACM.

  • Grinstead, C.M., & Snell, J.L. (1997). Introduction to probability, 2nd edn, AMS Bookstore, chap Central Limit Theorem, (pp. 325–360).

  • Hermoso, R., Centeno, R., Fasli, M. (2013). From blurry numbers to clear preferences: A mechanism to extract reputation in social networks. Expert Systems with Applications, 41 (5), 2269–2285.

    Article  Google Scholar 

  • Hill, W., Stead, L., Rosenstein, M., Furnas, G. (1995). Recommending and evaluating choices in a virtual community of use. Proceedings of the sigchi conference on human factors in computing systems. CHI ’95, (pp. 194–201). New York: ACM Press/Addison-Wesley Publishing Co.

  • Huynh, T.D., Jennings, N.R., Shadbolt, N.R. (2006). An integrated trust and reputation model for open multi-agent systems. Autonomous Agents and Multi-Agent Systems, 13 (2), 119–154.

    Article  Google Scholar 

  • Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20 (4), 422–446.

    Article  Google Scholar 

  • Jøsang, A., & Ismail, R. (2002). The beta reputation system. Proceedings of the 15th bled electronic commerce conference.

  • Jøsang, A., Ismail, R., Boyd, C. (2007). A survey of trust and reputation systems for online service provision. Decisions Support Systems, 43 (2), 618–644.

    Article  Google Scholar 

  • Kendall, M.G. (1938). A New Measure of Rank Correlation. Biometrika, 30 (1/2), 81+. doi: 10.2307/2332226.

    Article  Google Scholar 

  • Khosravifar, B., Bentahar, J., Gomrokchi, M., Alam, R. (2012). CRM: An efficient trust and reputation model for agent computing. Knowledge-Based Systems, 30 (0), 1 –16.

    Article  Google Scholar 

  • Kramer, M. (2007). Self-selection bias in reputation systems In Etalle, S., & Marsh, S. (Eds.), Trust management. IFIP International Federation for Information Processing (Vol. 238, pp. 255–268). Berlin Heildelberg New York: Springer.

  • Liu, Y., Li, K., Jin, Y., Zhang, Y., Qu, W. (2011). A novel reputation computation model based on subjective logic for mobile ad hoc networks. Future Generation Computer Systems, 27 (5), 547 –554.

    Article  Google Scholar 

  • Luca, M. (2011). Reviews, reputation, and revenue: The case of yelp.com. Harvard Business School Working Papers 12–016, Harvard Business School.

  • Negahban, S., Oh, S., Shah, D. (2012). Iterative ranking from pair-wise comparisons. Proceedings of the 26th annual conference on neural information processing systems, (pp. 2483–2491).

  • Pennock, D., Horvitz, E., Lawrence, S., Giles, C.L. (2000). Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. Proceedings of the sixteenth conference on uncertainty in artificial intelligence, (pp. 473–480). San Mateo: Morgan Kaufmann.

  • Pinyol, I., & Sabater-Mir, J. (2013). Computational trust and reputation models for open multi-agent systems: a review. Artificial Intelligence Review, 40 (1), 1–25.

    Article  Google Scholar 

  • Ramchurn, S.D., Huynh, D., Jennings, N.R. (2004). Trust in multi-agent systems. Knowledge Engineering Review, 19 (1), 1–25.

    Article  Google Scholar 

  • Resnick, P., Kuwabara, K., Zeckhauser, R., Friedman, E. (2000). Reputation systems. Communications of the ACM, 43 (12), 45–48.

    Article  Google Scholar 

  • Shearin, S., & Lieberman, H. (2001). Intelligent profiling by example. Proceedings of the 6th international conference on intelligent user interfaces. IUI ’01, (pp. 145–151). New York: ACM.

  • Sierra, C., & Sabater, J. (2005). Review on computionational trust and reputation models. Artificial Intelligence Review, 24 (1), 33 –60.

    Article  Google Scholar 

  • Sparling, E.I., & Sen, S. (2011). Rating: how difficult is it?. Proceedings of the 5th acm conference on recommender systems. RecSys ’11, (pp. 149–156). New York: ACM.

  • Tanimoto, T. (1958). An elementary mathematical theory of classification and prediction. IBM Internal Report.

  • Teacy, W.T.L., Patel, J., Jennings, N.R., Luck, M. (2006). Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems, 12 (2), 183–198.

    Article  Google Scholar 

  • Teacy, W.T.L., Luck, M., Rogers, A., Jennings, N.R. (2012). An efficient and versatile approach to trust and reputation using hierarchical bayesian modelling. Artificial Intelligence, 193, 149–185.

    Article  Google Scholar 

  • Yu, B., & Singh, M.P. (2002). An evidential model of distributed reputation management. AAMAS ’02: Proceedings of the first international joint conference on autonomous agents and multiagent systems, (pp. 294–301). New York: ACM Press.

  • Yu, B., Singh, M.P., Sycara, K. (2004). Developing trust in large-scale peer-to-peer systems. Proceedings of first IEEE symposium on multi-agent security and survivability, (pp. 1–10).

Download references

Acknowledgments

Work partially supported by the Spanish Ministry of Science and Innovation through the projects OVAMAH (grant TIN2009-13839-C03-02; co-funded by Plan E) and ”AT” (grant CSD2007-0022; CONSOLIDER-INGENIO 2010) and by the Spanish Ministry of Economy and Competitiveness through the project iHAS (grant TIN2012-36586-C03-02); and by the Region of Madrid through the eMadrid project (grant S2009-TIC1650).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Roberto Centeno.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Centeno, R., Hermoso, R. & Fasli, M. On the inaccuracy of numerical ratings: dealing with biased opinions in social networks. Inf Syst Front 17, 809–825 (2015). https://doi.org/10.1007/s10796-014-9526-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10796-014-9526-1

Keywords

Navigation