research-article

Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings

Authors:

Gediminas Adomavicius,

Jesse Bockstedt,

Jingjing ZhangAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 39, Issue 2

Article No.: 13, Pages 1 - 38

https://doi.org/10.1145/3430028

Published: 14 January 2021 Publication History

Abstract

Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias users’ preference ratings after item consumption in the direction of the predicted rating. Top-N lists represent another common approach for presenting item recommendations in recommender systems. Through three controlled laboratory experiments, we show that top-N lists do not induce a discernible bias in user preference judgments. This result is robust, holding for both lists of personalized item recommendations and lists of items that are top-rated based on averages of aggregate user ratings. Adding numerical ratings to the list items does generate a bias, consistent with earlier studies. Thus, in contexts where preference biases are of concern to an online retailer or platform, top-N lists, without numerical predicted ratings, would be a promising format for displaying item recommendations.

References

[1]

H. Abdollahpouri, R. Burke, and B. Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. In Proceedings of the 11th ACM Conference on Recommender Systems. ACM, pp. 42--46.

[2]

P. Adamopoulos and A. Tuzhilin. 2014. On over-specialization and concentration bias of recommendations: Probabilistic neighborhood selection in collaborative filtering systems. In Proceedings of the 8th ACM Conference on Recommender Systems (Foster City, Silicon Valley, Calif.). ACM, pp. 153--160.

[3]

G. Adomavicius, J. Bockstedt, S. Curley, and J. Zhang. 2013. Do recommender systems manipulate consumer preferences? A study of anchoring effects. Inf. Syst. Res. 24, 4 (2013), 956--975.

Digital Library

[4]

G. Adomavicius, J. Bockstedt, S. Curley, and J. Zhang. 2016. Effects of personalized versus aggregate ratings on consumer preference responses. In Proceedings of the Conference on Information Systems and Technology (Nashville, TN).

[5]

G. Adomavicius, J. Bockstedt, S. Curley, and J. Zhang. 2018. Effects of online recommendations on consumers’ willingness to pay. Inf. Syst. Res. 29, 1 (2018), 84--102.

Digital Library

[6]

G. Adomavicius, J. Bockstedt, S. Curley, and J. Zhang. 2019. Reducing recommender systems biases: An investigation of rating display designs. MIS Quarterly (MISQ) 43, 4 (2019), 1321--1341.

[7]

G. Adomavicius and Y. Kwon. 2012. Improving aggregate recommendation diversity using ranking-based techniques. IEEE Trans. Knowl. Data Eng. 24, 5 (2012), 896--911.

Digital Library

[8]

G. Adomavicius and A. Tuzhilin. 2005. Toward the next generation of recommendation system: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 6 (2005), 734--749.

Digital Library

[9]

A. Agarwal, K. Hosanagar, and M. D. Smith. 2011. Location, location, location: An analysis of profitability of position in online advertising markets. J. Market Res. 48, 6 (2011), 1057--1073.

[10]

X. Amatriain, J. M. Pujol, and N. Oliver. 2009. I like it... I like it not: Evaluating user ratings noise in recommender systems. In Proceedings of the International Conference on User Modeling, Adaptation, and Personalization. Springer, pp. 247--258.

[11]

S. Aral. 2014. The problem with online ratings. MIT Sloan Manage. Rev. 55, 2 (2014), 47.

[12]

R. M. Bell and Y. Koren. 2007. Improved neighborhood-based collaborative filtering. In Proceedings of KDD Cup'07, (San Jose, Calif.) ACM, New York, pp. 7--14.

[13]

T. Belluf, L. Xavier, and R. Giglio. 2012. Case study on the business value impact of personalized recommendations on a large online retailer. In Proceedings of the 6th ACM Conference on Recommender Systems (Dublin, Ireland) ACM, 277--280.

[14]

J. S. Breese, D. Heckerman, and C. Kadie. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (Madison, WI).

[15]

L. Chen, M. d. Gemmis, A. Felfernig, P. Lops, F. Ricci, and G. Semeraro. 2013. Human decision making and recommender systems. ACM Trans. Interact. Intell. Syst. 3, 3 (2013), Article 17.

Digital Library

[16]

D. Cosley, S. Lam, I. Albert, J. A. Konstan, and J. Riedl. 2003. Is seeing believing? how recommender interfaces affect users’ opinions. In Proceedings of the Conference on Human Factors in Computing Systems, (Fort Lauderdale, FL) ACM New York, pp585--592.

[17]

P. Cremonesi, Y. Koren, and R. Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the 4th ACM Conference on Recommender Systems (Barcelona, Spain) ACM, 39--46.

[18]

M. Deshpande and G. Karypis. 2004. Item-based top-n recommendation algorithms. ACM Trans. Inf. Syst. 22, 1 (2004), 143--177.

Digital Library

[19]

J. N. Druckman and C. D. Kam. 2011. Students as experimental participants. Cambridge Handbook of Experimental Political Science (1), 41--57.

[20]

D. Fleder and K. Hosanagar. 2009. Blockbuster culture's next rise or fall: The impact of recommender systems on sales diversity. Manage. Sci. 55, 5 (2009), 697--712.

Digital Library

[21]

A. Ghose, P. G. Ipeirotis, and B. Li. 2012. Designing ranking systems for hotels on travel search engines by mining user-generated and crowdsourced content. Market. Sci. 31, 3 (2012), 493--520.

Digital Library

[22]

K. Goldberg, T. Roeder, D. Gupta, and C. Perkins. 2001. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retriev. 4, 2 (2001), 133--151.

Digital Library

[23]

F. Guo and D. B. Dunson. 2015. Uncovering systematic bias in ratings across categories: A Bayesian approach. In Proceedings of the 9th ACM Conference on Recommender Systems (Vienna, Austria) ACM, pp. 317--320.

[24]

X. He, H. Zhang, M.-Y. Kan, and T.-S. Chua. 2016. Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) ACM, 549--558.

Digital Library

[25]

E. Hirt, F. Kardes, and K. Markman. 2004. Activating a mental simulation mind-set through generation of alternatives: Implications for debiasing in related and unrelated domains. J. Experi. Social Psych. 40, 3 (2004), 374--383.

[26]

K. Hosanagar, D. Fleder, D. Lee, and A. Buja. 2014. Will the global village fracture into tribes? Recommender systems and their effects on consumer fragmentation. Manage. Sci. 60, 4 (2014), 805--823.

[27]

M. S. Isaac and R. M. Schindler. 2014. The top-ten effect: Consumers’ subjective categorization of ranked lists. Journal of Consumer Research 40, 6 (2014), 1181--1202.

[28]

A. Jameson, M. C. Willemsen, A. Felfernig, M. de Gemmis, P. Lops, G. Semeraro, and L. Chen. 2015. Human decision making and recommender systems. In Recommender Systems Handbook, F. Ricci, L. Rokach and B. Shapira (eds.). Springer US, Boston, MA, pp. 611--648.

[29]

D. Jannach, L. Lerche, I. Kamehkhosh, and M. Jugovac. 2015. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25, 5, 1, 427--491.

[30]

K. Jerath, L. Ma, Y.-H. Park, and K. Srinivasan. 2011. A position paradox” in sponsored search auctions. Marketing Science 30, 4 (2011), 612--627.

Digital Library

[31]

T. Joachims, A. Swaminathan, and T. Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining, pp. 781--789.

[32]

S. Kabbur, X. Ning, and G. Karypis. 2013. Fism: Factored item similarity models for top-n recommender systems. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Chicago, Ill.), ACM, pp. 659--667.

[33]

Z. Kang, C. Peng, and Q. Cheng. 2016. Top-n recommender system via matrix completion. In Proceedings of the 30th AAAI Conference on Artificial Intelligence.

[34]

J. A. Konstan. 2004. Introduction to recommender systems: Algorithms and evaluation. ACM Trans. Inf, Syst. 22, 1 (2004), 1--4.

[35]

Y. Koren and R. Bell. 2015. Advances in collaborative filtering. In Recommender Systems Handbook, F. Ricci, L. Rokach and B. Shapira (eds.). Springer US, Boston, MA, 77--118.

[36]

S. Krishnan, J. Patel, M. J. Franklin, and K. Goldberg. 2014. A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. In Proceedings of the 8th ACM Conference on Recommender Systems. (Foster City, Silicon Valley, Calif.) ACM, pp137--144.

[37]

G. Linden, B. Smith, and J. York. 2003. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE CS 7, 1 (2003), 76--80.

Digital Library

[38]

B. Loni, R. Pagano, M. Larson, and A. Hanjalic. 2019. Top-N recommendation with multi-channel positive feedback using factorization machines. ACM Trans. Inf. Syst. 37, 2 (2019) Mar.

Digital Library

[39]

J. Lu, D. Wu, M. Mao, W. Wang, and G. Zhang. 2015. Recommender system application developments. Decis. Support Syst. 74, C (2015), 12--32.

[40]

W. Lu, F. L. Chung, W. H. Jiang, M. Ester, and W. Liu. 2019. A deep Bayesian tensor-based system for video recommendation. ACM Trans. Inf. Syst. 37, 1 (2019) Jan.

Digital Library

[41]

J. Monks and R. G. Ehrenberg. 1999. U.S. News & World Report's college rankings: Why they do matter. Change: The Magazine of Higher Learning 31, 6 (1999), 1999/11/01, 42--51.

[42]

T. Mussweiler and F. Strack. 1999. Hypothesis-consistent testing and semantic priming in the anchoring paradigm: A selective accessibility model. Journal of Experimental Social Psychology 35, 2 (1999), 136--164.

[43]

T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, and J. A. Konstan. 2014. Exploring the filter bubble: The effect of using recommender systems on content diversity. In Proceedings of the 23rd International Conference on World Wide Web. (Seoul, Korea) ACM, 677--686.

[44]

X. Ning and G. Karypis. 2012. Sparse linear methods with side information for top-n recommendations. In Proceedings of the 6th ACM Conference on Recommender Systems (Dublin, Ireland) ACM, 155--162.

[45]

E. Pariser. 2011. The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think. Penguin.

Digital Library

[46]

D. G. Pope. 2009. Reacting to rankings: Evidence from “America's Best Hospitals. J. Health Economics 28, 6 (2009), 1154--1165.

[47]

S. Prawesh and B. Padmanabhan. 2014. The “most popular news” recommender: count amplification and manipulation resistance. Inf. Syst. Res 25, 3 (2014), 569--589.

Digital Library

[48]

P. Resnick, R. K. Garrett, T. Kriplean, S. A. Munson, and N. J. Stroud. 2013. Bursting your (filter) bubble: Strategies for promoting diverse exposure. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work Companion (San Antonio, TX.) Association for Computing Machinery, 95--100.

[49]

P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the Conference on Computer Supported Cooperative Work, (Chapel Hill, NC), 175--186.

[50]

F. Ricci, L. Rokach, and B. Shapira. 2015. Recommender Systems Handbook. Springer.

[51]

Sa, #250, l. Vargas, and P. Castells. 2014. Improving sales diversity by recommending users to items. In Proceedings of the 8th ACM Conference on Recommender Systems (Foster City, Silicon Valley, Calif), ACM, 145--152.

[52]

B. Sarwar, G. Karypis, J. A. Konstan, and J. Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the WWW Conference (Hong Kong). ACM, 285--295.

[53]

T. Schnabel, A. Swaminathan, A. Singh, N. Chandak, and T. Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. Arxiv Preprint Arxiv:1602.05352.

[54]

S. Senecal and J. Nantel. 2004. The influence of online product recommendations on consumers’ online choices. J. Retal 80, 2 (2004), 2004/01/01/, 159--169.

[55]

B. Smith and G. Linden. 2017. Two decades of recommender systems at amazon. com. IEEE Internet Comput. 21, 3 (2017), 12--18.

Digital Library

[56]

J. B. Soll, K. L. Milkman, and J. W. Payne. 2016. A user's guide to debiasing. In The Wiley Blackwell Handbook of Judgment and Decision Making. John Wiley & Sons, Ltd, 924--951.

[57]

A. T. Sorensen. 2007. Bestseller lists and product variety. The Journal of Industrial Economics 55, 4 (2007), 715--738.

[58]

K. Truong, F. Ishikawa, and S. Honiden. 2007. Improving accuracy of recommender system by item clustering. IEICE Trans. on Information and Systems (E90D:9), Sep, 1363--1373.

[59]

A. Tversky and D. Kahneman. 1974. Judgment under uncertainty: heuristics and biases. Science 185, 4157 (1974), 1124--1131.

[60]

A. Tversky, S. Sattath, and P. Slovic. 1988. Contingent weighting in judgement and choice. Psychol. Review 95, 3 (1988), 371--384.

[61]

R. Ursu. 2017. The power of rankings: Quantifying the effect of rankings on online consumer search and purchase decisions. Market. Science.

[62]

M. Wan, J. Ni, R. Misra, and J. McAuley. 2020. Addressing marketing bias in product recommendations. In Proceedings of the 13th International Conference on Web Search and Data Mining, 618--626.

[63]

N. Xia and G. Karypis. 2011. Slim: Sparse linear methods for top-n recommender systems. In Proceedings of the 2011 IEEE 11th International Conference on Data Mining (ICDM), 497--506.

[64]

F. Xue, X. N. He, X. Wang, J. D. Xu, K. Liu, and R. C. Hong. 2019. Deep item-based collaborative filtering for top-n recommendation. ACM Trans. Inf. Syst. 37, 3 (2019) Jul.

Digital Library

[65]

L. Yang, Y. Cui, Y. Xuan, C. Wang, S. Belongie, and D. Estrin. 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems 279--287.

[66]

C. Y. Zhang, H. W. Liang, and K. Wang. 2016. Trip recommendation meets real-world constraints: Poi availability, diversity, and traveling time uncertainty. Acm Transactions on Information Systems 35, 1 (2016) Oct.

Digital Library

[67]

J. Zhang, G. Adomavicius, A. Gupta, and W. Ketter. 2019. Consumption and performance: Understanding longitudinal dynamics of recommender systems via an agent-based simulation framework. Information Systems Research.

[68]

T. Zhang and V. S. Iyengar. 2002. Recommender systems using linear classifiers. Journal of Machine Learning Research (2:Feb), 313--334.

Digital Library

[69]

X. Zhang, J. Zhao, and J. C. Lui. 2017. Modeling the assimilation-contrast effects in online product rating systems: Debiasing and recommendations. In Proceedings of the 11th ACM Conference on Recommender Systems, 98--106.

[70]

Z. Zolaktaf, R. Babanezhad, and R. Pottinger. 2018. A generic top-n recommendation framework for trading-off accuracy, novelty, and coverage. In Proceedings of the 2018 IEEE 34th International Conference on Data Engineering (ICDE): IEEE, 149--160.

Cited By

Awal GTehlan U(2024)Mapping the research landscape of recommender systems for digital librariesRecord and Library Journal10.20473/rlj.V10-I1.2024.180-19410:1(180-194)Online publication date: 22-Jun-2024
https://doi.org/10.20473/rlj.V10-I1.2024.180-194
Mousavi NAdamopoulos PBockstedt J(2023)The Decoy Effect and Recommendation SystemsInformation Systems Research10.1287/isre.2022.119734:4(1533-1553)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1287/isre.2022.1197
Zhang SJiang TKuang KFeng FYu JMa JZhao ZZhu JYang HChua TWu F(2023)SLED: Structure Learning based Denoising for RecommendationACM Transactions on Information Systems10.1145/361138542:2(1-31)Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1145/3611385
Show More Cited By

Index Terms

Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Presentation of retrieval results
    2. Retrieval tasks and goals
      1. Recommender systems

Recommendations

A framework for diversifying recommendation lists by user interest expansion

Recommender systems have been widely used to discover users' preferences and recommend interesting items to users during this age of information overload. Researchers in the field of recommender systems have realized that the quality of a top-N ...
User Personality and User Satisfaction with Recommender Systems

In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Serendipitous Personalized Ranking for Top-N Recommendation
WI-IAT '12: Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01

Serendipitous recommendation has benefitted both e-retailers and users. It tends to suggest items which are both unexpected and useful to users. These items are not only profitable to the retailers but also surprisingly suitable to consumers' tastes. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 39, Issue 2

April 2021

391 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/3444752

Editor:
Min Zhang
Tsinghua University, China

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 January 2021

Accepted: 01 October 2020

Revised: 01 August 2020

Received: 01 March 2020

Published in TOIS Volume 39, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
619
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Awal GTehlan U(2024)Mapping the research landscape of recommender systems for digital librariesRecord and Library Journal10.20473/rlj.V10-I1.2024.180-19410:1(180-194)Online publication date: 22-Jun-2024
https://doi.org/10.20473/rlj.V10-I1.2024.180-194
Mousavi NAdamopoulos PBockstedt J(2023)The Decoy Effect and Recommendation SystemsInformation Systems Research10.1287/isre.2022.119734:4(1533-1553)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1287/isre.2022.1197
Zhang SJiang TKuang KFeng FYu JMa JZhao ZZhu JYang HChua TWu F(2023)SLED: Structure Learning based Denoising for RecommendationACM Transactions on Information Systems10.1145/361138542:2(1-31)Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1145/3611385
Cai DQian SFang QHu JXu C(2023)User Cold-Start Recommendation via Inductive Heterogeneous Graph Neural NetworkACM Transactions on Information Systems10.1145/356048741:3(1-27)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1145/3560487
Wu HLong JLi NYu DNg M(2022)Adversarial Auto-encoder Domain Adaptation for Cold-start Recommendation with Positive and Negative HypergraphsACM Transactions on Information Systems10.1145/354410541:2(1-25)Online publication date: 21-Dec-2022
https://dl.acm.org/doi/10.1145/3544105
Afchar DMelchiorre ASchedl MHennequin REpure EMoussallam M(2022)Explainability in music recommender systemsAI Magazine10.1002/aaai.1205643:2(190-208)Online publication date: 23-Jun-2022
https://dl.acm.org/doi/10.1002/aaai.12056
Adomavicius GBockstedt JCurley SZhang J(2022)Recommender systems, ground truth, and preference pollutionAI Magazine10.1002/aaai.1205543:2(177-189)Online publication date: 23-Jun-2022
https://dl.acm.org/doi/10.1002/aaai.12055
Sonboli NBurke REkstrand MMehrotra R(2022)The multisided complexity of fairness in recommender systemsAI Magazine10.1002/aaai.1205443:2(164-176)Online publication date: 23-Jun-2022
https://dl.acm.org/doi/10.1002/aaai.12054

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents