skip to main content
10.1145/3488560.3498519acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article

The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?

Published: 15 February 2022 Publication History

Abstract

There has been sustained interest from both academia and industry throughout the years due to the importance and practicability of recommendation systems. However, several recent papers have pointed out critical issues with the evaluation process in recommender systems. Likewise, this paper takes an in-depth look at a fundamental but often neglected aspect of the evaluation procedure, i.e. the datasets themselves. To do so, we adopt a systematic and comprehensive approach to understand the datasets used for implicit feedback based top-K recommendation. We start by examining recent papers from top-tier conferences to find out how different datasets have been utilised thus far. Next, we look at the characteristics of these datasets to understand their similarities and differences. Finally, we conduct an empirical study to determine whether the choice of datasets used for evaluation can influence the observations and/or conclusions obtained. Our findings suggest that greater attention needs to be paid to the selection process of datasets used for evaluating recommender systems in order to improve the robustness of the obtained results.

Supplementary Material

MP4 File (WSDM22-fp798.mp4)
This is the video presentation for the paper titled "The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?". The choice of datasets used seems to be a fundamental but often neglected aspect, and we conduct a retrospective survey to gain a better understanding of recommendation datasets. For more information, please read the paper.

References

[1]
Gediminas Adomavicius and Jingjing Zhang. 2012. Impact of Data Characteristics on Recommender Systems Performance. ACM TMIS, Vol. 3, 1, Article 3 (April 2012), bibinfonumpages17 pages.
[2]
Rakesh Agrawal, Heikki Mannila, Ramakrishnan Srikant, Hannu Toivonen, and A. Inkeri Verkamo. 1996. Fast Discovery of Association Rules .American Association for Artificial Intelligence, USA, 307--328.
[3]
Roc'io Ca namares and Pablo Castells. 2017. A Probabilistic Reformulation of Memory-Based Collaborative Filtering: Implications on Popularity Biases. In SIGIR '17. 215--224.
[4]
Dong-Kyu Chae, Jihoo Kim, Duen Horng Chau, and Sang-Wook Kim. 2020. AR-CF: Augmenting Virtual Users and Items in Collaborative Filtering for Addressing Cold-Start Problems. In SIGIR '20. 1251--1260.
[5]
Chaochao Chen, Ziqi Liu, Peilin Zhao, Longfei Li, Jun Zhou, and Xiaolong Li. 2018. Distributed Collaborative Hashing and Its Applications in Ant Financial. In KDD '18 . 100--109.
[6]
Yihong Chen, Bei Chen, Xiangnan He, Chen Gao, Yong Li, Jian-Guang Lou, and Yue Wang. 2019. $łambda$Opt: Learn to Regularize Recommender Models in Finer Levels. In KDD '19 . 978--986.
[7]
Evangelia Christakopoulou and George Karypis. 2016. Local Item-Item Models For Top-N Recommendation. In RecSys '16. 67--74.
[8]
Evangelia Christakopoulou and George Karypis. 2018. Local Latent Space Models for Top-N Recommendation. In KDD '18 . 1235--1243.
[9]
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. In RecSys '19. 101--109.
[10]
Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, and Felice Antonio Merra. 2020. How Dataset Characteristics Affect the Robustness of Collaborative Recommendation Models. In SIGIR '20. 951--960.
[11]
Travis Ebesu, Bin Shen, and Yi Fang. 2018. Collaborative Memory Network for Recommendation Systems. In SIGIR'18 . 515--524.
[12]
Ehtsham Elahi, Wei Wang, Dave Ray, Aish Fenton, and Tony Jebara. 2019. Variational Low Rank Multinomials for Collaborative Filtering with Side-Information. In RecSys '19 . 340--347.
[13]
Chen Gao, Chao Huang, Dongsheng Lin, Depeng Jin, and Yong Li. 2020. DPLCF: Differentially Private Local Collaborative Filtering. In SIGIR '20 . 961--970.
[14]
Corrado Gini. 1921. Measurement of Inequality of Incomes. The Economic Journal, Vol. 31, 121 (1921), 124--126.
[15]
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR '20. 639--648.
[16]
Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. 2018. Adversarial Personalized Ranking for Recommendation. In SIGIR'18 . 355--364.
[17]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW '17. 173--182.
[18]
Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast Matrix Factorization for Online Recommendation with Implicit Feedback. In SIGIR'16 . 549--558.
[19]
Jonathan L. Herlocker, Joseph A. Konstan, Al Borchers, and John Riedl. 1999. An Algorithmic Framework for Performing Collaborative Filtering. In SIGIR '99 . 230--237.
[20]
Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In ICDM '08 . 263--272.
[21]
Shuyi Ji, Yifan Feng, Rongrong Ji, Xibin Zhao, Wanwan Tang, and Yue Gao. 2020. Dual Channel Hypergraph Collaborative Filtering. In KDD '20. 2020--2029.
[22]
Jyun-Yu Jiang, Patrick H. Chen, Cho-Jui Hsieh, and Wei Wang. 2020. Clustering and Constructing User Coresets to Accelerate Large-Scale Top-K Recommender Systems. In WWW '20. 2177--2187.
[23]
Manas R. Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K. Adams, Pranav Khaitan, Jiahui Liu, and Quoc V. Le. 2020. Neural Input Search for Large Scale Recommendation Models. In KDD '20 . 2387--2397.
[24]
Farhan Khawar, Leonard Poon, and Nevin L. Zhang. 2020. Learning the Structure of Auto-Encoding Recommenders. In WWW '20 . 519--529.
[25]
Walid Krichene and Steffen Rendle. 2020. On Sampled Metrics for Item Recommendation. In KDD '20. 1748--1757.
[26]
Dongsheng Li, Chao Chen, Qin Lv, Hansu Gu, Tun Lu, Li Shang, Ning Gu, and Stephen M. Chu. 2018. AdaError: An Adaptive Learning Rate Method for Matrix Approximation-Based Collaborative Filtering. In WWW '18. 741--751.
[27]
Defu Lian, Haoyu Wang, Zheng Liu, Jianxun Lian, Enhong Chen, and Xing Xie. 2020. LightRec: A Memory and Search-Efficient Recommender System. In WWW '20. 695--705.
[28]
Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. 2016. Modeling User Exposure in Recommendation. In WWW '16. 951--961.
[29]
Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In WWW '18 . 689--698.
[30]
Chenghao Liu, Tao Lu, Xin Wang, Zhiyong Cheng, Jianling Sun, and Steven C.H. Hoi. 2019 a. Compositional Coding for Collaborative Filtering. In SIGIR'19. 145--154.
[31]
Feng Liu, Huifeng Guo, Xutao Li, Ruiming Tang, Yunming Ye, and Xiuqiang He. 2020 a. End-to-End Deep Reinforcement Learning Based Recommendation with Supervised Embedding. In WSDM '20. 384--392.
[32]
Huafeng Liu, Liping Jing, Jingxuan Wen, Zhicheng Wu, Xiaoyi Sun, Jiaqi Wang, Lin Xiao, and Jian Yu. 2020 b. Deep Global and Local Generative Model for Recommendation. In WWW '20 . 551--561.
[33]
Huafeng Liu, Jingxuan Wen, Liping Jing, and Jian Yu. 2019 b. Deep Generative Ranking for Personalized Recommendation. In RecSys '19 . 34--42.
[34]
Ramon Lopes, Renato Assuncc ao, and Rodrygo L.T. Santos. 2016. Efficient Bayesian Methods for Graph-Based Recommendation. In RecSys '16 . 333--340.
[35]
Chen Ma, Liheng Ma, Yingxue Zhang, Ruiming Tang, Xue Liu, and Mark Coates. 2020. Probabilistic Metric Learning with Adaptive Margin for Top-K Recommendation. In KDD '20 . 1036--1044.
[36]
Khalil Muhammad, Qinqin Wang, Diarmuid O'Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. 2020. FedFast: Going Beyond Average for Faster Training of Federated Recommender Systems. In KDD '20 . 1234--1242.
[37]
Athanasios N. Nikolakopoulos, Dimitris Berberidis, George Karypis, and Georgios B. Giannakis. 2019. Personalized Diffusions for Top-n Recommendation. In RecSys '19. 260--268.
[38]
Athanasios N. Nikolakopoulos and George Karypis. 2019. RecWalk: Nearly Uncoupled Random Walks for Top-N Recommendation. In WSDM '19 . 150--158.
[39]
Rasaq Otunba, Raimi A. Rufai, and Jessica Lin. 2017. MPR: Multi-Objective Pairwise Ranking. In RecSys '17. 170--178.
[40]
Bibek Paudel, Fabian Christoffel, Chris Newell, and Abraham Bernstein. 2016. Updatable, Accurate, Diverse, and Scalable Recommendations for Interactive Applications. ACM TiiS, Vol. 7, 1, Article 1 (Dec. 2016), bibinfonumpages34 pages.
[41]
Shameem A. Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. 2016. A Coverage-Based Approach to Recommendation Diversity On Similarity Graph. In RecSys '16 . 15--22.
[42]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-Based Collaborative Filtering Recommendation Algorithms. In WWW '01 . 285--295.
[43]
Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I. Nikolenko. 2020. RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback. In WSDM '20. 528--536.
[44]
Shaoyun Shi, Weizhi Ma, Min Zhang, Yongfeng Zhang, Xinxing Yu, Houzhi Shan, Yiqun Liu, and Shaoping Ma. 2020. Beyond User Embedding Matrix: Learning to Hash for Modeling Large-Scale Users in Recommendation. In SIGIR '20. 319--328.
[45]
Harald Steck, Maria Dimakopoulou, Nickolai Riabov, and Tony Jebara. 2020. ADMM SLIM: Sparse Recommendations for Many Users. In WSDM '20. 555--563.
[46]
Jianing Sun, Wei Guo, Dengcheng Zhang, Yingxue Zhang, Florence Regol, Yaochen Hu, Huifeng Guo, Ruiming Tang, Han Yuan, Xiuqiang He, and Mark Coates. 2020 a. A Framework for Recommending Accurate and Diverse Items Using Bayesian Graph Convolutional Neural Networks. In KDD '20. 2030--2039.
[47]
Jianing Sun, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, Xiuqiang He, Chen Ma, and Mark Coates. 2020 c. Neighbor Interaction Aware Graph Convolution Networks for Recommendation. In SIGIR '20 . 1289--1298.
[48]
Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020 b. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. In RecSys '20. 23--32.
[49]
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent Relational Metric Learning via Memory-Based Attention for Collaborative Ranking. In WWW '18 . 729--739.
[50]
Thanh Tran, Xinyue Liu, Kyumin Lee, and Xiangnan Kong. 2019. Signed Distance-Based Deep Memory Recommender. In WWW '19. 1841--1852.
[51]
Lucas Vinh Tran, Yi Tay, Shuai Zhang, Gao Cong, and Xiaoli Li. 2020. HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems. In WSDM '20. 609--617.
[52]
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural Graph Collaborative Filtering. In SIGIR'19. 165--174.
[53]
Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. 2020. Disentangled Graph Collaborative Filtering. In SIGIR '20. 1001--1010.
[54]
Ga Wu, Maksims Volkovs, Chee Loong Soon, Scott Sanner, and Himanshu Rai. 2019. Noise Contrastive Estimation for One-Class Collaborative Filtering. In SIGIR'19 . 135--144.
[55]
Yao Wu, Christopher DuBois, Alice X. Zheng, and Martin Ester. 2016a. Collaborative Denoising Auto-Encoders for Top-N Recommender Systems. In WSDM '16 . 153--162.
[56]
Yao Wu, Xudong Liu, Min Xie, Martin Ester, and Qing Yang. 2016b. CCCF: Improving Collaborative Filtering via Scalable User-Item Co-Clustering. In WSDM '16 . 73--82.
[57]
Hanwang Zhang, Fumin Shen, Wei Liu, Xiangnan He, Huanbo Luan, and Tat-Seng Chua. 2016. Discrete Collaborative Filtering. In SIGIR'16. 325--334.
[58]
Yuan Zhang, Xiaoran Xu, Hanning Zhou, and Yan Zhang. 2020. Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation. In WSDM '20. 735--743.
[59]
Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, and Philip S. Yu. 2018. Spectral Collaborative Filtering. In RecSys '18. 311--319.

Cited By

View all
  • (2025)Latent diffusion model-based data poisoning attack against QoS-aware cloud API recommender systemComputer Networks10.1016/j.comnet.2025.111120(111120)Online publication date: Feb-2025
  • (2025)Enhancing session-based trip recommendations using matrix factorization: a study on algorithm efficiency and resource utilizationThe Journal of Supercomputing10.1007/s11227-024-06726-181:1Online publication date: 1-Jan-2025
  • (2024)Characteristics of the Learning Data of a Session-Based Recommendation System and their Impact on the Performance of the SystemProceedings of the 32nd International Conference on Information Systems Development10.62036/ISD.2024.24Online publication date: 2024
  • Show More Cited By

Index Terms

  1. The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
    February 2022
    1690 pages
    ISBN:9781450391320
    DOI:10.1145/3488560
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 February 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. data characteristics
    2. datasets
    3. evaluation
    4. item recommendation

    Qualifiers

    • Research-article

    Funding Sources

    • Ministry of Education Singapore

    Conference

    WSDM '22

    Acceptance Rates

    Overall Acceptance Rate 498 of 2,863 submissions, 17%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)172
    • Downloads (Last 6 weeks)18
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Latent diffusion model-based data poisoning attack against QoS-aware cloud API recommender systemComputer Networks10.1016/j.comnet.2025.111120(111120)Online publication date: Feb-2025
    • (2025)Enhancing session-based trip recommendations using matrix factorization: a study on algorithm efficiency and resource utilizationThe Journal of Supercomputing10.1007/s11227-024-06726-181:1Online publication date: 1-Jan-2025
    • (2024)Characteristics of the Learning Data of a Session-Based Recommendation System and their Impact on the Performance of the SystemProceedings of the 32nd International Conference on Information Systems Development10.62036/ISD.2024.24Online publication date: 2024
    • (2024)Our Model Achieves Excellent Performance on MovieLens: What Does It Mean?ACM Transactions on Information Systems10.1145/367516342:6(1-25)Online publication date: 18-Oct-2024
    • (2024)Group Validation in Recommender Systems: Framework for Multi-layer Performance EvaluationACM Transactions on Recommender Systems10.1145/36408202:1(1-25)Online publication date: 7-Mar-2024
    • (2024)Informed Dataset Selection with ‘Algorithm Performance Spaces’Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3691704(1085-1090)Online publication date: 8-Oct-2024
    • (2024)Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential RecommendationsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688195(1067-1072)Online publication date: 8-Oct-2024
    • (2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
    • (2024)FPSR+: Toward Robust, Efficient, and Scalable Collaborative Filtering With Partition-Aware Item Similarity ModelingIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.341808036:12(8283-8296)Online publication date: Dec-2024
    • (2024)Context-aware cross feature attentive network for click-through rate predictionsApplied Intelligence10.1007/s10489-024-05659-954:19(9330-9344)Online publication date: 13-Jul-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media