review-article

Report on the workshop on reproducibility and replication in recommender systems evaluation (RepSys)

Authors:

Alejandro Bellogín,

Pablo Castells,

Domonkos TikkAuthors Info & Claims

ACM SIGIR Forum, Volume 48, Issue 1

Pages 29 - 35

https://doi.org/10.1145/2641383.2641389

Published: 26 June 2014 Publication History

Abstract

Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems. When an experiment is repeated by a different researcher and exactly the same result is obtained, we can say the experiment has been replicated. When the results are not exactly the same but the conclusions are compatible with the prior ones, we have a reproduction of the experiment. Reproducibility and replication involve recommendation algorithm implementations, experimental protocols, and evaluation metrics. While the problem of reproducibility and replication has been recognized in the Recommender Systems community, the need for a clear solution remains largely unmet, which motivates the main questions addressed in the present workshop.

References

[1]

Joeran Beel, Marcel Genzmehr, Stefan Langer, Andreas Nürnberger, and Bela Gipp. A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In Proceedings of the International Workshop on Re- producibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 7--14, New York, NY, USA, 2013. ACM.

Digital Library

[2]

Joeran Beel, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, and Andreas Nürnberger. Research paper recommender system evaluation: A quantitative literature survey. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 15--22, New York, NY, USA, 2013. ACM.

Digital Library

[3]

Alejandro Bellogín, Pablo Castells, Alan Said, and Domonkos Tikk, editors. RepSys '13: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, New York, NY, USA, 2013. ACM.

Digital Library

[4]

Thierry Bertin-Mahieux, Daniel P. W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In ISMIR, pages 591--596, 2011.

[5]

Jim Blomo, Martin Ester, and Marty Field. Recsys challenge 2013. In RecSys, pages 489--490, 2013.

Digital Library

[6]

Gideon Dror, Noam Koenigstein, Yehuda Koren, and Markus Weimer. The yahoo! music dataset and kdd-cup '11. In KDD Cup, pages 8--18, 2012.

[7]

Reid Holmes, Martin P. Robillard, Robert J. Walker, and Thomas Zimmermann. Rsse 2010: Second international workshop on recommendation systems for software engineering. In ICSE (2), pages 455--456, 2010.

Digital Library

[8]

Joseph A. Konstan and Gediminas Adomavicius. Toward identification and adoption of best practices in algorithmic recommender systems research. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 23--28, New York, NY, USA, 2013. ACM.

Digital Library

[9]

Andrej Koýir, Ante Odić, and Marko Tkalčič. How to improve the statistical power of the 10-fold cross validation scheme in recommender systems. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 3--6, New York, NY, USA, 2013. ACM.

Digital Library

[10]

Mark Levy. Offline evaluation of recommender systems: All pain and no gain? In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 1--1, New York, NY, USA, 2013. ACM.

Digital Library

[11]

Benjamin M. Marlin, Richard S. Zemel, Sam T. Roweis, and Malcolm Slaney. Collaborative filtering and the missing at random assumption. In Ronald Parr and Linda C. van der Gaag, editors, UAI, pages 267--275. AUAI Press, 2007.

[12]

Martin P. Robillard, Walid Maalej, Robert J. Walker, and Thomas Zimmermann, editors. Recommendation Systems in Software Engineering. Springer, 2014.

Digital Library

[13]

Guy Shani and Asela Gunawardana. Evaluating recommendation systems. In Recommender Systems Handbook, pages 257--297. 2011.

[14]

Sören Sonnenburg, Mikio L. Braun, Cheng Soon Ong, Samy Bengio, Léon Bottou, Geoffrey Holmes, Yann LeCun, Klaus-Robert Möller, Fernando Pereira, Carl Edward Rasmussen, Gunnar Rätsch, Bernhard Schölkopf, Alexander J. Smola, Pascal Vincent, Jason Weston, and Robert C. Williamson. The need for open source software in machine learning. Journal of Machine Learning Research, 8:2443--2466, 2007.

Digital Library

[15]

Andrew Trotman, Charles L. A. Clarke, Iadh Ounis, Shane Culpepper, Marc-Allen Cartright, and Shlomo Geva. Open source information retrieval: a report on the SIGIR 2012 workshop. SIGIR Forum, 46(2):95--101, 2012.

Digital Library

[16]

Max L. Wilson, Ed H. Chi, David Coyle, and Paul Resnick, editors. Proceedings of the CHI 2013 Workshop on the Replication of HCI Research, Paris, France, April 27-28, 2013, volume 976 of CEUR Workshop Proceedings. CEUR-WS.org, 2013.

[17]

Wai Gen Yee, Michel Beigbeder, andWray L. Buntine. SIGIR06 workshop report: Open source information retrieval systems (OSIR06). SIGIR Forum, 40(2):61--65, 2006.

Digital Library

Cited By

Polatidis NPapaleonidas APimenidis EIliadis L(2020)An explanation-based approach for experiment reproducibility in recommender systemsNeural Computing and Applications10.1007/s00521-019-04274-x32:16(12259-12266)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s00521-019-04274-x
Polatidis NPimenidis EFish AKapetanakis S(2019)A Guideline-Based Approach for Assisting with the Reproducibility of Experiments in Recommender Systems EvaluationInternational Journal on Artificial Intelligence Tools10.1142/S021821301960011X28:08(1960011)Online publication date: 2-Dec-2019
https://doi.org/10.1142/S021821301960011X
Polatidis NPimenidis E(2018)Reproduction of Experiments in Recommender Systems Evaluation Based on ExplanationsEngineering Applications of Neural Networks10.1007/978-3-319-98204-5_16(194-200)Online publication date: 27-Jul-2018
https://doi.org/10.1007/978-3-319-98204-5_16
Show More Cited By

Index Terms

Report on the workshop on reproducibility and replication in recommender systems evaluation (RepSys)
1. General and reference

Recommendations

Workshop on reproducibility and replication in recommender systems evaluation: RepSys
RecSys '13: Proceedings of the 7th ACM conference on Recommender systems

Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems. When an experiment is repeated by a different researcher and exactly the same result is ...
Towards reproducibility in recommender-systems research

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system ...
RepSys '13: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation

Comments

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum

ACM SIGIR Forum Volume 48, Issue 1

June 2014

42 pages

ISSN:0163-5840

DOI:10.1145/2641383

Editors:
Fernando Diaz
Microsoft Research NYC, New York, NY
,
Raman Chandrasekar
Serials Solutions, Seattle, WA

Issue’s Table of Contents

Copyright © 2014 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2014

Published in SIGIR Volume 48, Issue 1

Check for updates

Qualifiers

Review-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Polatidis NPapaleonidas APimenidis EIliadis L(2020)An explanation-based approach for experiment reproducibility in recommender systemsNeural Computing and Applications10.1007/s00521-019-04274-x32:16(12259-12266)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s00521-019-04274-x
Polatidis NPimenidis EFish AKapetanakis S(2019)A Guideline-Based Approach for Assisting with the Reproducibility of Experiments in Recommender Systems EvaluationInternational Journal on Artificial Intelligence Tools10.1142/S021821301960011X28:08(1960011)Online publication date: 2-Dec-2019
https://doi.org/10.1142/S021821301960011X
Polatidis NPimenidis E(2018)Reproduction of Experiments in Recommender Systems Evaluation Based on ExplanationsEngineering Applications of Neural Networks10.1007/978-3-319-98204-5_16(194-200)Online publication date: 27-Jul-2018
https://doi.org/10.1007/978-3-319-98204-5_16
Polatidis NKapetanakis SPimenidis EKosmidis K(2018)Reproducibility of Experiments in Recommender Systems EvaluationArtificial Intelligence Applications and Innovations10.1007/978-3-319-92007-8_34(401-409)Online publication date: 22-May-2018
https://doi.org/10.1007/978-3-319-92007-8_34
Cremonesi PSaid ATikk DZhou M(2016)Introduction to the Special Issue on Recommender System BenchmarkingACM Transactions on Intelligent Systems and Technology10.1145/28706277:3(1-4)Online publication date: 8-Mar-2016
https://dl.acm.org/doi/10.1145/2870627
Beel JBreitinger CLanger SLommatzsch AGipp B(2016)Towards reproducibility in recommender-systems researchUser Modeling and User-Adapted Interaction10.1007/s11257-016-9174-x26:1(69-101)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1007/s11257-016-9174-x

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents