Skip to main content
SpringerLink
Log in

Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14163))

Abstract

We describe the first edition of the LongEval CLEF 2023 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.mturk.com/.

  2. 2.

    https://clef-longeval.github.io/.

  3. 3.

    https://codalab.lisn.upsaclay.fr/competitions/12762.

References

  1. Alkhalifa, R., et al.: Longeval: longitudinal evaluation of model performance at CLEF 2023. In: Kamps, J., et al. (eds.) ECIR 2023. Lecture Notes in Computer Science, vol. 13982, pp. 499–505. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_58

    Chapter  Google Scholar 

  2. Alkhalifa, R., Kochkina, E., Zubiaga, A.: Opinions are made to be changed: temporally adaptive stance classification. In: Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks, pp. 27–32 (2021)

    Google Scholar 

  3. Alkhalifa, R., Kochkina, E., Zubiaga, A.: Building for tomorrow: Assessing the temporal persistence of text classifiers. arXiv preprint arXiv:2205.05435 (2022)

  4. Alkhalifa, R., Zubiaga, A.: Capturing stance dynamics in social media: open challenges and research directions. Int. J. Digit. Hum. 3, 1–21 (2022)

    Google Scholar 

  5. Galuščáková, P., et al.: Longeval-retrieval: French-English dynamic test collection for continuous web search evaluation (2023)

    Google Scholar 

  6. Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., Camacho-collados, J.: TimeLMs: diachronic language models from Twitter. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 251–260. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-demo.25, https://aclanthology.org/2022.acl-demo.25

  7. Urbano, J., Lima, H., Hanjalic, A.: A new perspective on score standardization. In: International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1061–1064 (2019)

    Google Scholar 

  8. Yin, W., Alkhalifa, R., Zubiaga, A.: The emojification of sentiment on social media: Collection and analysis of a longitudinal Twitter sentiment dataset. arXiv preprint arXiv:2108.13898 (2021)

Download references

Acknowledgements

This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1) and The Alan Turing Institute (grant no. EP/N510129/1) through project funding and its Enrichment PhD Scheme for Iman Bilal. This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

Author information

Authors and Affiliations

  1. Queen Mary University of London, London, UK

    Rabab Alkhalifa, Elena Kochkina, Maria Liakata & Arkaitz Zubiaga

  2. Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

    Rabab Alkhalifa

  3. University of Warwick, Coventry, UK

    Iman Bilal & Maria Liakata

  4. Cardiff University, Cardiff, UK

    Hsuvas Borkakoty, Jose Camacho-Collados, Luis Espinosa-Anke & Daniel Loureiro

  5. Alan Turing Institute, London, UK

    Elena Kochkina & Maria Liakata

  6. Qwant, Paris, France

    Romain Deveaud & Christophe Servan

  7. Univ. Grenoble Alpes, CNRS, Grenoble INP (Institute of Engineering Univ. Grenoble Alpes.), LIG, Grenoble, France

    Gabriela Gonzalez-Saez, Petra Galuščáková, Lorraine Goeuriot & Philippe Mulhem

  8. University of Bath, Bath, UK

    Harish Tayyar Madabushi

  9. Research Studios Austria, Data Science Studio, Vienna, Austria

    Alaa El-Ebshihy & Florina Piroi

  10. Charles University, Prague, Czech Republic

    Martin Popel

  11. Paris-Saclay University, CNRS, LISN, Orsay, France

    Christophe Servan

  12. AMPLYFI, Cardiff, UK

    Luis Espinosa-Anke

Authors
  1. Rabab Alkhalifa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iman Bilal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hsuvas Borkakoty
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jose Camacho-Collados
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Romain Deveaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alaa El-Ebshihy
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Luis Espinosa-Anke
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gabriela Gonzalez-Saez
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Petra Galuščáková
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lorraine Goeuriot
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Elena Kochkina
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Maria Liakata
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Daniel Loureiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Philippe Mulhem
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Florina Piroi
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Martin Popel
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Christophe Servan
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Harish Tayyar Madabushi
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Arkaitz Zubiaga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philippe Mulhem .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. CERTH-ITI, Thessaloniki, Greece

    Theodora Tsikrika

  4. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Utrecht University, Utrecht, The Netherlands

    Anastasia Giachanou

  6. Elsevier, Amsterdam, The Netherlands

    Dan Li

  7. University of Amsterdam, Amsterdam, The Netherlands

    Mohammad Aliannejadi

  8. University of Lausanne, Lausanne, Switzerland

    Michalis Vlachos

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alkhalifa, R. et al. (2023). Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42448-9_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42447-2

  • Online ISBN: 978-3-031-42448-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation