skip to main content
10.1145/3460231.3474244acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

User Bias in Beyond-Accuracy Measurement of Recommendation Algorithms

Published: 13 September 2021 Publication History

Abstract

There are various biases in recommender systems. Recognizing biases, as well as unfairness caused by problematic biases, is the first step of system optimization. Related studies on algorithmic biases are mainly from the perspective of either items or users. For the latter (we call it “algorithmic user bias”), existing works have considered algorithms’ accuracy performances measured by accuracy metrics like RMSE. However, algorithmic user biases in beyond-accuracy measurements have rarely been studied, even though beyond-accuracy oriented recommendation algorithms have been increasingly investigated, with the purpose of breaking through the personalization limits of traditional accuracy-oriented algorithms (such as the typical “filter bubble” phenomenon). To fill in the research gap, in this work, we employ a large-scale survey dataset collected from a commercial platform, in which more than 11,000 users’ ratings on the recommendation’s 5 performance objectives (i.e., relevance, diversity, novelty, unexpectedness, and serendipity) and 8 kinds of user characteristics (i.e., gender, age, big-5 personality traits, and curiosity) are available. We study user biases of four algorithms (i.e., HOT, Rel-CF, Nov-CF, and Ser-CF) in terms of those five measurements between user groups of the eight user characteristics. We further look into users’ behavior patterns like the preference of using more positive ratings, in order to interpret the observed biases. Finally, based on the observed algorithmic user bias and users’ behavior patterns, we analyze the possible factors leading to the biases and recognize problematic biases that may lead to unfairness.

Supplementary Material

MP4 File (RecSys2021_WangandChen_v2.mp4)
Presentation video

References

[1]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2019. The Unfairness of Popularity Bias in Recommendation. arxiv:1907.13286 [cs.IR]
[2]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, and Edward Malthouse. 2021. User-Centered Evaluation of Popularity Bias in Recommender Systems. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 11 pages. https://doi.org/10.1145/3450613.3456821
[3]
Panagiotis Adamopoulos and Alexander Tuzhilin. 2015. On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected. ACM Transactions on Intelligent Systems and Technology 5, 4 (Jan. 2015), 1–32. https://doi.org/10.1145/2559952
[4]
Gediminas Adomavicius and YoungOk Kwon. 2012. Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques. IEEE Transactions on Knowledge and Data Engineering 24, 5 (May 2012), 896–911. https://doi.org/10.1109/TKDE.2011.15
[5]
Ricardo Baeza-Yates. 2018. Bias on the Web. Commun. ACM 61, 6 (May 2018), 54–61. https://doi.org/10.1145/3209581
[6]
Toine Bogers and Lennart Björneborn. 2013. Micro-serendipity: Meaningful Coincidences in Everyday Life Shared on Twitter. In Proceedings of the iConference 2013. 196–208. https://doi.org/10.9776/13175
[7]
Ludovico Boratto, Gianni Fenu, and Mirko Marras. 2019. The Effect of Algorithmic Bias on Recommender Systems for Massive Open Online Courses. In Advances in Information Retrieval(Lecture Notes in Computer Science), Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra(Eds.). Springer International Publishing, Cham, 457–472. https://doi.org/10.1007/978-3-030-15712-8_30
[8]
Ludovico Boratto, Gianni Fenu, and Mirko Marras. 2021. Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation. Information Processing & Management 58, 1 (Jan. 2021), 102387. https://doi.org/10.1016/j.ipm.2020.102387
[9]
China Internet Network Information Center. 2018. The 41st Statistical Report on Internet Development in China. http://www.cac.gov.cn/files/pdf/cnnic/CNNIC41.pdf
[10]
Li Chen, Yonghua Yang, Ningxia Wang, Keping Yang, and Quan Yuan. 2019. How Serendipity Improves User Satisfaction with Recommendations? A Large-Scale User Evaluation. In The World Wide Web Conference(WWW ’19). Association for Computing Machinery, New York, NY, USA, 240–250. https://doi.org/10.1145/3308558.3313469
[11]
Anup A. Deshmukh, Pratheeksha Nair, and Shrisha Rao. 2018. A Scalable Clustering Algorithm for Serendipity in Recommender Systems. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW). 1279–1288. https://doi.org/10.1109/ICDMW.2018.00182
[12]
Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness. In Conference on Fairness, Accountability and Transparency. PMLR, 172–186. http://proceedings.mlr.press/v81/ekstrand18b.html
[13]
Donald E. Farrar and Robert R. Glauber. 1967. Multicollinearity in Regression Analysis: The Problem Revisited. The Review of Economics and Statistics 49, 1 (1967), 92–107.
[14]
Samuel D. Gosling, Peter J. Rentfrow, and William B. Swann. 2003. A Very Brief Measure of the Big-Five Personality Domains. Journal of Research in Personality 37, 6 (Dec. 2003), 504–528. https://doi.org/10.1016/S0092-6566(03)00046-1
[15]
Asela Gunawardana and Guy Shani. 2015. Evaluating Recommendation Systems. Springer US, Boston, MA, 265–308.
[16]
Rong Hu and Pearl Pu. 2014. Exploring Personality’s Effect on Users’ Rating Behavior. In CHI ’14 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’14). Association for Computing Machinery, New York, NY, USA, 2599–2604. https://doi.org/10.1145/2559206.2581317
[17]
Dawn Iacobucci, Steven S. Posavac, Frank R. Kardes, Matthew J. Schneider, and Deidre L. Popovich. 2015. The Median Split: Robust, Refined, and Revived. Journal of Consumer Psychology 25, 4 (2015), 690–704. https://doi.org/10.1016/j.jcps.2015.06.014
[18]
Dietmar Jannach, Lukas Lerche, Iman Kamehkhosh, and Michael Jugovac. 2015. What Recommenders Recommend: An Analysis of Recommendation Biases and Possible Countermeasures. 25, 5 (2015), 427–491. https://doi.org/10.1007/s11257-015-9165-3
[19]
Marius Kaminskas and Derek Bridge. 2014. Measuring Surprise in Recommender Systems. In Proceedings of the Workshop on Recommender Systems Evaluation: Dimensions and Design (Workshop Programme of the 8th ACM Conference on Recommender Systems). Silicon Valley, United States, 6.
[20]
Marius Kaminskas and Derek Bridge. 2016. Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems. ACM Trans. Interact. Intell. Syst. 7, 1 (Dec. 2016), 2:1–2:42. https://doi.org/10.1145/2926720
[21]
Todd B. Kashdan, Matthew W. Gallagher, Paul J. Silvia, Beate P. Winterstein, William E. Breen, Daniel Terhar, and Michael F. Steger. 2009. The Curiosity and Exploration Inventory-II: Development, Factor Structure, and Psychometrics. Journal of Research in Personality 43, 6 (Dec. 2009), 987–998. https://doi.org/10.1016/j.jrp.2009.04.011
[22]
Denis Kotkov, Shuaiqiang Wang, and Jari Veijalainen. 2016. A Survey of Serendipity in Recommender Systems. Knowledge-Based Systems 111 (2016), 180–192. https://doi.org/10.1016/j.knosys.2016.08.014
[23]
Dominik Kowald, Markus Schedl, and Elisabeth Lex. 2020. The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study. In Advances in Information Retrieval (Cham) (Lecture Notes in Computer Science), Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, and Flávio Martins (Eds.). Springer International Publishing, 35–42. https://doi.org/10.1007/978-3-030-45442-5_5
[24]
Pan Li, Maofei Que, Zhichao Jiang, Yao Hu, and Alexander Tuzhilin. 2020. PURS: Personalized Unexpected Recommender System for Improving User Satisfaction. In Proceedings of the 14th ACM Conference on Recommender Systems. Association for Computing Machinery, 10. https://doi.org/10.1145/3383313.3412238
[25]
Xueqi Li, Wenjun Jiang, Weiguang Chen, Jie Wu, and Guojun Wang. 2019. HAES: A New Hybrid Approach for Movie Recommendation with Elastic Serendipity. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19). Association for Computing Machinery, New York, NY, USA, 1503–1512. https://doi.org/10.1145/3357384.3357868
[26]
Xueqi Li, Wenjun Jiang, Weiguang Chen, Jie Wu, Guojun Wang, and Kenli Li. 2020. Directional and Explainable Serendipity Recommendation. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 122–132. https://doi.org/10.1145/3366423.3380100
[27]
George Loewenstein. 1994. The Psychology of Curiosity: A Review and Reinterpretation.Psychological Bulletin 116, 1 (1994), 75.
[28]
Qiuxia Lu, Tianqi Chen, Weinan Zhang, Diyi Yang, and Yong Yu. 2012. Serendipitous Personalized Ranking for Top-N Recommendation. In 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology. IEEE, Macau, China, 258–265. https://doi.org/10.1109/WI-IAT.2012.135
[29]
Valentina Maccatrozzo, Manon Terstall, Lora Aroyo, and Guus Schreiber. 2017. SIRUP: Serendipity In Recommendations via User Perceptions. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (Limassol, Cyprus) (IUI ’17). Association for Computing Machinery, New York, NY, USA, 35–44. https://doi.org/10.1145/3025171.3025185
[30]
Henry B. Mann and Donald R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50 – 60. https://doi.org/10.1214/aoms/1177730491
[31]
Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback Loop and Bias Amplification in Recommender Systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Virtual Event, Ireland) (CIKM ’20). Association for Computing Machinery, New York, NY, USA, 2145–2148. https://doi.org/10.1145/3340531.3412152
[32]
Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2021. Flatter Is Better: Percentile Transformations for Recommender Systems. ACM Transactions on Intelligent Systems and Technology 12, 2 (March 2021), 19:1–19:16. https://doi.org/10.1145/3437910
[33]
Christian Matt, Thomas Hess, Alexander Benlian, and Christian Weiß. 2014. Escaping from the Filter Bubble? The Effects of Novelty and Serendipity on Users’ Evaluations of Online Recommendations. In Proceedings of the 35th International Conference on Information Systems (Auckland, New Zealand) (ICIS).
[34]
Robert R. McCrae and Oliver P. John. 1992. An Introduction to the Five-Factor Model and Its Applications. Journal of personality 60, 2 (1992), 175–215.
[35]
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A Survey on Bias and Fairness in Machine Learning. arxiv:1908.09635 [cs] http://arxiv.org/abs/1908.09635
[36]
Alessandro B. Melchiorre, Eva Zangerle, and Markus Schedl. 2020. Personality Bias of Music Recommendation Algorithms. In Fourteenth ACM Conference on Recommender Systems (Virtual Event Brazil). Association for Computing Machinery, New York, NY, USA, 533–538. https://doi.org/10.1145/3383313.3412223
[37]
Aurora Mobile. 2018. 2017 Online Shopping App Market Research Report. https://community.jiguang.cn/article/246360
[38]
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019. Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. Frontiers in Big Data 2(2019). https://doi.org/10.3389/fdata.2019.00013
[39]
Shameem A. Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. 2016. A Coverage-Based Approach to Recommendation Diversity On Similarity Graph. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM Press, Boston, Massachusetts, USA, 15–22. https://doi.org/10.1145/2959100.2959149
[40]
Markus Schedl, David Hauger, Katayoun Farrahi, and Marko Tkalčič. 2015. On the Influence of User Characteristics on Music Recommendation Algorithms. In Advances in Information Retrieval (Cham) (Lecture Notes in Computer Science), Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr (Eds.). Springer International Publishing, 339–345. https://doi.org/10.1007/978-3-319-16354-3_37
[41]
Harini Suresh and John V. Guttag. 2020. A Framework for Understanding Unintended Consequences of Machine Learning. (Feb. 2020). arxiv:1901.10002 [cs, stat]
[42]
Marko Tkalcic and Li Chen. 2015. Personality and Recommender Systems. In Recommender systems handbook. Springer, 715–739.
[43]
Horace Walpole. 1960. To Mann, Monday 18 January 1754. In Horace Walpole’s Correspondence. Yale University Press, 407–411.
[44]
Ningxia Wang, Li Chen, and Yonghua Yang. 2020. The Impacts of Item Features and User Characteristics on Users’ Perceived Serendipity of Recommendations. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (Genoa, Italy) (UMAP ’20). Association for Computing Machinery, 266–274. https://doi.org/10.1145/3340631.3394863
[45]
Shanfeng Wang, Maoguo Gong, Haoliang Li, and Junwei Yang. 2016. Multi-Objective Optimization for Long Tail Recommendation. Knowledge-Based Systems 104 (July 2016), 145–155. https://doi.org/10.1016/j.knosys.2016.04.018
[46]
Wikipedia contributors. 2021. Liebig’s Law of The Minimum — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Liebig%27s_law_of_the_minimum&oldid=1002543189 [Online; accessed 28-April-2021].
[47]
Wen Wu, Li Chen, and Yu Zhao. 2018. Personalizing recommendation diversity based on user personality. User Modeling and User-Adapted Interaction 28, 3 (2018), 237–276.
[48]
Pengfei Zhao and Dik Lun Lee. 2016. How Much Novelty is Relevant?: It Depends on Your Curiosity. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR ’16). Association for Computing Machinery, New York, NY, USA, 315–324. https://doi.org/10.1145/2911451.2911488
[49]
Ziwei Zhu, Yun He, Xing Zhao, Yin Zhang, Jianling Wang, and James Caverlee. 2021. Popularity-Opportunity Bias in Collaborative Filtering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (Virtual Event, Israel) (WSDM ’21). Association for Computing Machinery, New York, NY, USA, 85–93. https://doi.org/10.1145/3437963.3441820

Cited By

View all
  • (2024)Explaining Recommendation Fairness from a User/Item PerspectiveACM Transactions on Information Systems10.1145/369887743:1(1-30)Online publication date: 5-Oct-2024
  • (2024)Fair Reciprocal Recommendation in Matching MarketsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688130(209-218)Online publication date: 8-Oct-2024
  • (2024)Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender SystemsACM Transactions on Information Systems10.1145/363786942:4(1-32)Online publication date: 9-Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '21: Proceedings of the 15th ACM Conference on Recommender Systems
September 2021
883 pages
ISBN:9781450384582
DOI:10.1145/3460231
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Recommender systems
  2. algorithmic bias
  3. beyond-accuracy objectives
  4. curiosity
  5. fairness
  6. personality
  7. serendipity
  8. user bias

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Hong Kong Research Grants Council

Conference

RecSys '21: Fifteenth ACM Conference on Recommender Systems
September 27 - October 1, 2021
Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)123
  • Downloads (Last 6 weeks)7
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Explaining Recommendation Fairness from a User/Item PerspectiveACM Transactions on Information Systems10.1145/369887743:1(1-30)Online publication date: 5-Oct-2024
  • (2024)Fair Reciprocal Recommendation in Matching MarketsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688130(209-218)Online publication date: 8-Oct-2024
  • (2024)Understanding or Manipulation: Rethinking Online Performance Gains of Modern Recommender SystemsACM Transactions on Information Systems10.1145/363786942:4(1-32)Online publication date: 9-Feb-2024
  • (2024)RADio* – An Introduction to Measuring Normative Diversity in News RecommendationsACM Transactions on Recommender Systems10.1145/36364653:1(1-29)Online publication date: 2-Aug-2024
  • (2024)Evaluating Group Fairness in News Recommendations: A Comparative Study of Algorithms and MetricsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664897(337-346)Online publication date: 27-Jun-2024
  • (2023)Recommender Systems and Competition on Subscription-Based PlatformsSSRN Electronic Journal10.2139/ssrn.4428125Online publication date: 2023
  • (2023)Deep Learning Models for Serendipity Recommendations: A Survey and New PerspectivesACM Computing Surveys10.1145/360514556:1(1-26)Online publication date: 26-Aug-2023
  • (2023)Bias and Debias in Recommender System: A Survey and Future DirectionsACM Transactions on Information Systems10.1145/356428441:3(1-39)Online publication date: 7-Feb-2023
  • (2023)A Survey on the Fairness of Recommender SystemsACM Transactions on Information Systems10.1145/354733341:3(1-43)Online publication date: 7-Feb-2023
  • (2023)Filter bubbles in recommender systems: Fact or fallacy—A systematic reviewWIREs Data Mining and Knowledge Discovery10.1002/widm.151213:6Online publication date: 3-Aug-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media