demonstration

RepSys: Framework for Interactive Evaluation of Recommender Systems

Authors:

Jan Šafařík,

Vojtěch Vančura,

Pavel KordíkAuthors Info & Claims

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

Pages 636 - 639

https://doi.org/10.1145/3523227.3551469

Published: 13 September 2022 Publication History

Abstract

Making recommender systems more transparent and auditable is crucial for the future adoption of these systems. Available tools typically present mostly errors of models aggregated over all test users, which is often insufficient to uncover hidden biases and problems. Moreover, the emphasis is primarily on the accuracy of recommendations but less on other important metrics, such as the diversity of recommended items, the extent of catalog coverage, or the opportunity to discover novel items at bestsellers’ expense. In this work, we propose RepSys, a framework for evaluating recommender systems. Our work offers a set of highly interactive approaches for investigating various scenario recommendations, analyzing a dataset, and evaluating distributions of various metrics that combine visualization techniques with existing offline evaluation methods. RepSys framework is available under an open-source license to other researchers.

Supplementary Material

MP4 File (repsys_no_login_no_music_1080p.mp4)

Narrated screen capture of RepSys in action.

Download
112.75 MB

References

[1]

Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2018. Popularity-Aware Item Weighting for Long-Tail Recommendation. ArXiv preprint arXiv:1802.05382(2018).

[2]

Akshay Agrawal, Alnur Ali, and Stephen Boyd. 2021. Minimum-Distortion Embedding. ArXiv preprint arXiv:2103.02559(2021).

[3]

Vito Walter Anelli, Alejandro Bellogin, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021-07-11. Elliot. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval(2021-07-11), 2405–2414. https://doi.org/10.1145/3404835.3463245

Digital Library

[4]

Pablo Castells, Neil J. Hurley, and Saul Vargas. 2015. Novelty and Diversity in Recommender Systems. Recommender Systems Handbook(2015), 881–918. https://doi.org/10.1007/978-1-4899-7637-6_26

[5]

Ludovik Coba, Roberto Confalonieri, and Markus Zanker. 2022. RecoXplainer. IEEE Computational Intelligence Magazine 17, 1 (2022), 46–58. https://doi.org/10.1109/MCI.2021.3129958

Digital Library

[6]

Michael D. Ekstrand, Ben Carterette, and Fernando Diaz. 2021. Evaluating Recommenders with Distributions. RecSys Workshop on Perspectives on Evaluation (2021).

[7]

Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John T. Riedl. 2011. Rethinking the recommender research ecosystem. Proceedings of the fifth ACM conference on Recommender systems - RecSys ’11 (2011), 133–140.

Digital Library

[8]

Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy. Proceedings of the fourth ACM conference on Recommender systems - RecSys ’10 (2010), 257–260. https://doi.org/10.1145/1864708.1864761

Digital Library

[9]

Asela Gunawardana and Guy Shani. 2009. A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. The Journal of Machine Learning Research 10, 1 (2009), 2935–2962. https://dl.acm.org/doi/10.1145/963770.963772

Digital Library

[10]

Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec. In UMAP Workshops, Vol. 4. Citeseer.

[11]

Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. 2020. DeepRecSys. 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) (2020), 982–995. https://doi.org/10.1109/ISCA45697.2020.00084

Digital Library

[12]

F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages. https://doi.org/10.1145/2827872

Digital Library

[13]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4 (2002), 422–446. https://doi.org/10.1145/582415.582418

Digital Library

[14]

Antoine Ledent, Rodrigo Alves, and Marius Kloft. 2021. Orthogonal inductive matrix completion. IEEE Transactions on Neural Networks and Learning Systems (2021).

[15]

Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW ’18 (2018), 689–698. https://doi.org/10.1145/3178876.3186150

Digital Library

[16]

Leland McInnes. c2018. Umap-learn: Uniform Manifold Approximation and Projection for Dimension Reduction. https://umap-learn.readthedocs.io

[17]

Karl Pearson. 1901. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science (1901), 559–572.

[18]

Plotly c2021. Plotly JavaScript Open Source Graphing Library. https://plotly.com/javascript

[19]

Vargas Saúl. 2015. Novelty and diversity evaluation and enhancement in recommender systems. PhD thesis. Universidad Autónoma de Madrid, Spain, Madrid.

[20]

Barry Smyth and Paul McClave. 2001-7-12. Similarity vs. Diversity. Case-Based Reasoning Research and Development (2001-7-12), 347–361. https://doi.org/10.1007/3-540-44593-5_25

[21]

Nasim Sonboli, Masoud Mansoury, Ziyue Guo, Shreyas Kadekodi, Weiwen Liu, Zijun Liu, Andrew Schwartz, and Robin Burke. 2021-10-26. Librec-auto. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. ACM, New York, NY, USA, 4584–4593. https://doi.org/10.1145/3459637.3482006

Digital Library

[22]

Harald Steck. 2013-10-12. Evaluation of recommendations. Proceedings of the 7th ACM conference on Recommender systems (2013-10-12), 213–220. https://doi.org/10.1145/2507157.2507160

Digital Library

[23]

Harald Steck. 2019. Embarrassingly Shallow Autoencoders for Sparse Data. The World Wide Web Conference on - WWW ’19(2019), 3251–3257. https://doi.org/10.1145/3308558.3313710

Digital Library

[24]

Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020-09-22. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. Fourteenth ACM Conference on Recommender Systems (2020-09-22), 23–32. https://doi.org/10.1145/3383313.3412489

Digital Library

[25]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605.

[26]

Vojtěch Vančura and Pavel Kordík. 2021. Deep Variational Autoencoder with Shallow Parallel Path for Top-N Recommendation (VASP). Artificial Neural Networks and Machine Learning – ICANN 2021 (2021), 138–149. https://doi.org/10.1007/978-3-030-86383-8_11

Digital Library

[27]

Zygmunt Zajac. 2017. Goodbooks-10k: a new dataset for book recommendations. http://fastml.com/goodbooks-10k. FastML (2017).

Cited By

Vančura V(2023)Scalable and Explainable Linear Shallow Autoencoders for Collaborative Filtering from Industrial PerspectiveProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3595630(290-295)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3595630

Index Terms

RepSys: Framework for Interactive Evaluation of Recommender Systems
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Learning latent representations
  2. Modeling and simulation
    1. Model development and analysis
      1. Model verification and validation
2. Human-centered computing
  1. Visualization
    1. Visualization systems and tools
      1. Visualization toolkits

Recommendations

Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03

Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
A Novel Framework for Improving Recommender Diversity
International Workshop on Behavior and Social Informatics on Behavior and Social Computing - Volume 8178

Recommender systems are being used to assist users in finding relevant items from a large set of alternatives in many online applications. However, while most research up to this point has focused on improving the accuracy of recommender systems, other ...
A Scalable, Accurate Hybrid Recommender System
WKDD '10: Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data Mining

Recommender systems apply machine learning techniques for filtering unseen information and can predict whether a user would like a given resource. There are three main types of recommender systems: collaborative filtering, content-based filtering, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

September 2022

743 pages

ISBN:9781450392785

DOI:10.1145/3523227

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2022

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Funding Sources

Grant Agency of the Czech Technical University

Conference

RecSys '22

Sponsor:

RecSys '22: Sixteenth ACM Conference on Recommender Systems

September 18 - 23, 2022

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
314
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)10

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Vančura V(2023)Scalable and Explainable Linear Shallow Autoencoders for Collaborative Filtering from Industrial PerspectiveProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3595630(290-295)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3595630

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten