Skip to main content

Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11018))

  • 1120 Accesses

Abstract

Reproducibility has become increasingly important for many research areas, among those IR is not an exception and has started to be concerned with reproducibility and its impact on research results. This paper describes our first attempt to propose a lab on reproducibility named CENTRE and held during CLEF 2018. The aim of CENTRE is to run a reproducibility challenge across all the major IR evaluation campaigns and to provide the IR community with a venue where previous research results can be explored and discussed. This paper reports the participant results and preliminary considerations on the first edition of CENTRE@CLEF 2018, as well as some suggestions for future editions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.acm.org/publications/policies/artifact-review-badging.

  2. 2.

    http://www.centre-eval.org/clef2018/.

  3. 3.

    https://bitbucket.org/centre_eval/c2018_dataintelligence/src/master/.

References

  1. Agirre, E., Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2008: ad hoc track overview. In: Peters, C., et al. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 15–37. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04447-2_2

    Chapter  Google Scholar 

  2. Arguello, J., Crane, M., Diaz, F., Lin, J., Trotman, A.: Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). SIGIR Forum 49(2), 107–116 (2015)

    Article  Google Scholar 

  3. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Has adhoc retrieval improved since 1994? In: Allan, J., Aslam, J.A., Sanderson, M., Zhai, C., Zobel, J. (eds.) Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), pp. 692–693. ACM Press, New York (2009)

    Google Scholar 

  4. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Cheung, D.W.L., Song, I.Y., Chu, W.W., Hu, X., Lin, J.J. (eds.) Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)

    Google Scholar 

  5. Braschler, M.: CLEF 2003 – overview of results. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 44–63. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30222-3_5

    Chapter  Google Scholar 

  6. Clarke, C.L.A., Craswell, N., Voorhees, E.M.: Overview of the TREC 2012 web track. In: Voorhees, E.M., Buckland, L.P. (eds.) Proceedings of the Twenty-First Text REtrieval Conference (TREC 2012), pp. 1–8. National Institute of Standards and Technology (NIST), Special Publication 500–298, Washington (2013)

    Google Scholar 

  7. Collins-Thompson, K., Diaz, F., Clarke, C.L.A., Voorhees, E.M.: TREC 2013 web track overview. In: Voorhees, E.M. (ed.) Proceedings of the Twenty-Second Text REtrieval Conference (TREC 2013). National Institute of Standards and Technology (NIST), Special Publication 500–302, Washington (2014)

    Google Scholar 

  8. Collins-Thompson, K., Macdonald, C., Bennett, P.N., Voorhees, E.M.: TREC 2014 web track overview. In: Voorhees, E.M., Ellis, A. (eds.) Proceedings of the Twenty-Third Text REtrieval Conference (TREC 2014). National Institute of Standards and Technology (NIST), Special Publication 500–308, Washington (2015)

    Google Scholar 

  9. Di Nunzio, G.M., Ferro, N., Jones, G.J.F., Peters, C.: CLEF 2005: ad hoc track overview. In: Peters, C., et al. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 11–36. Springer, Heidelberg (2006). https://doi.org/10.1007/11878773_2

    Chapter  Google Scholar 

  10. Ferro, N.: Reproducibility challenges in information retrieval evaluation. ACM J. Data Inf. Qual. (JDIQ) 8(2), 8:1–8:4 (2017)

    Google Scholar 

  11. Ferro, N., Fuhr, N., Järvelin, K., Kando, N., Lippold, M., Zobel, J.: Increasing reproducibility in IR: findings from the Dagstuhl seminar on “reproducibility of data-oriented experiments in e-science". SIGIR Forum 50(1), 68–82 (2016)

    Article  Google Scholar 

  12. Ferro, N., Kelly, D.: SIGIR initiative to implement ACM artifact review and badging. SIGIR Forum 52(1) (2018)

    Google Scholar 

  13. Ferro, N., Peters, C.: CLEF 2009 ad hoc track overview: TEL and Persian tasks. In: Peters, C., et al. (eds.) CLEF 2009. LNCS, vol. 6241, pp. 13–35. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15754-7_2

    Chapter  Google Scholar 

  14. Freire, J., Fuhr, N., Rauber, A. (eds.): Report from Dagstuhl seminar 16041: reproducibility of data-oriented experiments in e-science. Dagstuhl Reports, vol. 6, no 1. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany (2016)

    Google Scholar 

  15. Gallagher, L., Mackenzie, J., Benham, R., Chen, R.C., Scholer, F., Culpepper, J.S.: RMIT at the NTCIR-13 we want web task. In: Kando, N., Fujita, S., Kato, M.P., Manabe, T. (eds.) Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies, pp. 402–406. National Institute of Informatics, Tokyo (2017)

    Google Scholar 

  16. Guyot, J., Radhouani, S., Falquet, G.: Ontology-based multilingual information retrieval. In: Peters, C., Quochi, V., Ferro, N. (eds.) CLEF 2005 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2005). http://ceur-ws.org/Vol-1171/. ISSN 1613–0073

  17. Jungwirth, M., Hanbury, A.: Replicating an experiment in cross-lingual information retrieval with explicit semantic analysis. In: Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.) CLEF 2018 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2018). ISSN 1613–0073

    Google Scholar 

  18. Kendall, M.G.: Rank Correlation Methods. Griffin, Oxford (1948)

    MATH  Google Scholar 

  19. Kenney, J.F., Keeping, E.S.: Mathematics of Statistics - Part One, 3rd edn. D. Van Nostrand Company, Princeton (1954)

    MATH  Google Scholar 

  20. Kharazmi, S., Scholer, F., Vallet, D., Sanderson, M.: Examining additivity and weak baselines. ACM Trans. Inf. Syst. (TOIS) 34(4), 23:1–23:18 (2016)

    Article  Google Scholar 

  21. Lin, J., et al.: Toward reproducible baselines: the open-source IR reproducibility challenge. In: Ferro, N., et al. (eds.) ECIR 2016. LNCS, vol. 9626, pp. 408–420. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30671-1_30

    Chapter  Google Scholar 

  22. Luo, C., Sakai, T., Liu, Y., Dou, Z., Xiong, C., Xu, J.: Overview of the NTCIR-13 we want web task. In: Kando, N., Fujita, S., Kato, M.P., Manabe, T. (eds.) Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies, pp. 394–401. National Institute of Informatics, Tokyo (2017)

    Google Scholar 

  23. McCreadie, R., et al.: University of Glasgow at TREC 2014: experiments with terrier in contextual suggestion, temporal summarisation and web tracks. In: Voorhees, E.M., Ellis, A. (eds.) Proceedings of the Twenty-Third Text REtrieval Conference (TREC 2014). National Institute of Standards and Technology (NIST), Special Publication 500–308, Washington (2015)

    Google Scholar 

  24. Nguyen, D., Overwijk, A., Hauff, C., Trieschnigg, D.R.B., Hiemstra, D., de Jong, F.: WikiTranslate: query translation for cross-lingual information retrieval using only Wikipedia. In: Peters, C., et al. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 58–65. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04447-2_6

    Chapter  Google Scholar 

  25. Savoy, J.: Report on CLEF-2003 multilingual tracks. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 64–73. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30222-3_6

    Chapter  Google Scholar 

  26. Sorg, P., Cimiano, P.: Cross-lingual information retrieval with explicit semantic analysis. In: Borri, F., Nardi, A., Peters, C., Ferro, N. (eds.) CLEF 2008 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2008). http://ceur-ws.org/Vol-1174/. ISSN 1613–0073

  27. Yang, P., Fang, H.: Evaluating the effectiveness of axiomatic approaches in web track. In: Voorhees, E.M. (ed.) Proceedings of the Twenty-Second Text REtrieval Conference (TREC 2013). National Institute of Standards and Technology (NIST), Special Publication 500–302, Washington (2014)

    Google Scholar 

  28. Zobel, J., Webber, W., Sanderson, M., Moffat, A.: Principles for robust evaluation infrastructure. In: Agosti, M., Ferro, N., Thanos, C. (eds.) Proceedings of the Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011), pp. 3–6. ACM Press, New York (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Maria Maistro .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ferro, N., Maistro, M., Sakai, T., Soboroff, I. (2018). Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm. In: Bellot, P., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2018. Lecture Notes in Computer Science(), vol 11018. Springer, Cham. https://doi.org/10.1007/978-3-319-98932-7_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-98932-7_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-98931-0

  • Online ISBN: 978-3-319-98932-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics