Abstract
Reproducibility has become increasingly important for many research areas, among those IR is not an exception and has started to be concerned with reproducibility and its impact on research results. This paper describes our second attempt to propose a lab on reproducibility named CENTRE, held during CLEF 2019. The aim of CENTRE is to run both a replicability and reproducibility challenge across all the major IR evaluation campaigns and to provide the IR community with a venue where previous research results can be explored and discussed. This paper reports the participant results and preliminary considerations on the second edition of CENTRE@CLEF 2019.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Allan, J., et al.: Research frontiers in information retrieval – report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018). In: SIGIR Forum, vol. 52, no. 1, June 2018
Allan, J., Harman, D.K., Kanoulas, E., Li, D., Van Gysel, C., Voorhees, E.M.: TREC 2017 common core track overview. In: Voorhees and Ellis [28]
Allan, J., Harman, D.K., Kanoulas, E., Li, D., Van Gysel, C., Voorhees, E.M.: TREC 2018 common core track overview. In: Voorhees and Ellis [29]
Arguello, J., Crane, M., Diaz, F., Lin, J., Trotman, A.: Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). In: SIGIR Forum, vol. 49, no. 2, pp. 107–116 (2015)
Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Has adhoc retrieval improved since 1994? In: Allan, J., Aslam, J.A., Sanderson, M., Zhai, C., Zobel, J. (eds.) Proceedings of 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), pp. 692–693. ACM Press, New York (2009)
Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Cheung, D.W.L., Song, I.Y., Chu, W.W., Hu, X., Lin, J.J. (eds.) Proceedings of 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)
Benham, R., et al.: RMIT at the 2018 TREC CORE Track. In: Voorhees and Ellis [29]
Breuer, T., Schaer, P.: Replicability and reproducibility of automatic routing runs. In: Cappellato et al. [9]
Cappellato, L., Ferro, N., Losada, D., Müller, H. (eds.): CLEF 2019 Working Notes, CEUR Workshop Proceedings. CEUR-WS.org (2019)
Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): CLEF 2018 Working Notes, CEUR Workshop Proceedings. CEUR-WS.org, ISSN 1613–0073 (2018)
Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., Wu, Z.Z.: The SIGIR 2019 open-source IR replicability challenge (OSIRRC 2019). In: Chevalier, M., Gaussier, É., Piwowarski, B., Maarek, Y., Nie, J.Y., Scholer, F. (eds.) Proceedings of 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019) (2019)
Ferro, N., et al.: The Dagstuhl perspectives workshop on performance modeling and prediction. In: SIGIR Forum, vol. 52, no. 1, June 2018
Ferro, N., Fuhr, N., Järvelin, K., Kando, N., Lippold, M., Zobel, J.: Increasing reproducibility in IR: findings from the Dagstuhl seminar on “reproducibility of data-oriented experiments in e-science”. In: SIGIR Forum, vol. 50, no. 1, pp. 68–82 (2016)
Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2019: overview of the replicability and reproducibility tasks. In: Cappellato et al. [9]
Ferro, N., Kelly, D.: SIGIR initiative to implement ACM artifact review and badging. In: SIGIR Forum, vol. 52, no. 1, June 2018
Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2018: overview of the replicability task. In: Cappellato et al. [10]
Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: Overview of CENTRE@CLEF 2018: a first tale in the systematic reproducibility realm. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 239–246. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_23
Freire, J., Fuhr, N., Rauber, A. (eds.): Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. Dagstuhl Reports, vol. 6, no. 1. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany (2016)
Fuhr, N.: Some common mistakes in IR evaluation, and how they can be avoided. In: SIGIR Forum vol. 51, no. 3, pp. 32–41 (2017)
Grossman, M.R., Cormack, G.V.: MRG\(\_\)UWaterloo and WaterlooCormack participation in the TREC 2017 common core track. In: Voorhees and Ellis [28]
Jungwirth, M., Hanbury, A.: Replicating an experiment in cross-lingual information retrieval with explicit semantic analysis. In: Cappellato et al. [10]
Kendall, M.G.: Rank Correlation Methods. Griffin, Oxford (1948)
Kenney, J.F., Keeping, E.S.: Mathematics of Statistics - Part One, 3rd edn. D. Van Nostrand Company, Princeton (1954)
Kharazmi, S., Scholer, F., Vallet, D., Sanderson, M.: Examining additivity and weak baselines. ACM Trans. Inf. Syst. (TOIS) 34(4), 23:1–23:18 (2016)
Lin, J., et al.: Toward reproducible baselines: the open-source IR reproducibility challenge. In: Ferro, N., et al. (eds.) ECIR 2016. LNCS, vol. 9626, pp. 408–420. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30671-1_30
Munafò, M.R., et al.: A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021:1–0021:9 (2017)
Sakai, T., Ferro, N., Soboroff, I., Zeng, Z., Xiao, P., Maistro, M.: Overview of the NTCIR-14 CENTRE task. In: Ishita, E., Kando, N., Kato, M.P., Liu, Y. (eds.) Proceedings of 14th NTCIR Conference on Evaluation of Information Access Technologies, pp. 494–509. National Institute of Informatics, Tokyo (2019)
Voorhees, E.M., Ellis, A. (eds.): The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). National Institute of Standards and Technology (NIST), Special Publication 500–324, Washington, USA (2018)
Voorhees, E.M., Ellis, A. (eds.): The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018). National Institute of Standards and Technology (NIST), Washington, USA (2019)
Zobel, J., Webber, W., Sanderson, M., Moffat, A.: Principles for robust evaluation infrastructure. In: Agosti, M., Ferro, N., Thanos, C. (eds.) Proceedings of Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011), pp. 3–6. ACM Press, New York (2011)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Soboroff, I. (2019). Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In: Crestani, F., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2019. Lecture Notes in Computer Science(), vol 11696. Springer, Cham. https://doi.org/10.1007/978-3-030-28577-7_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-28577-7_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28576-0
Online ISBN: 978-3-030-28577-7
eBook Packages: Computer ScienceComputer Science (R0)