Abstract
This paper introduces the planned second LongEval Lab, part of the CLEF 2024 conference. The aim of the lab’s two tasks is to give researchers test data for addressing temporal effectiveness persistence challenges in both information retrieval and text classification, motivated by the fact that model performance degrades as the test data becomes temporally distant from the training data. LongEval distinguishes itself from traditional IR and classification tasks by emphasizing the evaluation of models designed to mitigate performance drop over time using evolving data. The second LongEval edition will further engage the IR community and NLP researchers in addressing the crucial challenge of temporal persistence in models, exploring the factors that enable or hinder it, and identifying potential solutions along with their limitations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Qwant being mostly used by French speaker, it explains why it is easier to gather data (user queries and documents) in this language rather than English.
- 2.
Qwant search engine: https://www.qwant.com/.
- 3.
- 4.
- 5.
References
Alkhalifa, R., et al.: Extended overview of the CLEF-2023 longeval lab on longitudinal evaluation of model performance (2023). https://api.semanticscholar.org/CorpusID:259953335
Alkhalifa, R., Kochkina, E., Zubiaga, A.: Opinions are made to be changed: temporally adaptive stance classification. In: Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks, pp. 27–32 (2021)
Alkhalifa, R., Kochkina, E., Zubiaga, A.: Building for tomorrow: assessing the temporal persistence of text classifiers. Inf. Process. Manag. 60(2), 103200 (2023)
Alkhalifa, R., Zubiaga, A.: Capturing stance dynamics in social media: open challenges and research directions. Int. J. Digital Humanit. 1–21 (2022)
Chapelle, O., Zhang, Y.: A dynamic bayesian network click model for web search ranking. In: Proceedings of the 18th International Conference on World Wide Web, pp. 1–10. WWW 2009, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1526709.1526711
Chuklin, A., Markov, I., Rijke, M.D.: Click models for web search. Synth. Lect. Inf. Concepts Retrieval Serv. 7(3), 1–115 (2015). https://doi.org/10.2200/S00654ED1V01Y201507ICR043
Effrosynidis, D., Karasakalidis, A.I., Sylaios, G., Arampatzis, A.: The climate change twitter dataset. Expert Syst. Appl. 204, 117541 (2022). https://doi.org/10.1016/j.eswa.2022.117541, https://www.sciencedirect.com/science/article/pii/S0957417422008624
Florio, K., Basile, V., Polignano, M., Basile, P., Patti, V.: Time of your hate: the challenge of time in hate speech detection on social media. Appl. Sci. 10(12), 4180 (2020)
Küçük, D., Can, F.: Stance detection: a survey. ACM Comput. Surv. 53(1), 1–37 (2020). https://doi.org/10.1145/3369026
Lukes, J., Søgaard, A.: Sentiment analysis under temporal shift. In: Proceedings of the 9th workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 65–71 (2018)
Mohammad, S.M., Sobhani, P., Kiritchenko, S.: Stance and sentiment in tweets. ACM Trans. Internet Technol. 17(3), 1–23 (2017). https://doi.org/10.1145/3003433, http://alt.qcri.org/semeval2016/task6/
Ren, R., et al.: A thorough examination on zero-shot dense retrieval (2022). arxiv:2204.12755). https://doi.org/10.48550/ARXIV.2204.12755, https://arxiv.org/abs/2204.12755
Acknowledgements
This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1). This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2023062) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2023062 LINDAT/CLARIAH-CZ.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Alkhalifa, R. et al. (2024). LongEval: Longitudinal Evaluation of Model Performance at CLEF 2024. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14613. Springer, Cham. https://doi.org/10.1007/978-3-031-56072-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-56072-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56071-2
Online ISBN: 978-3-031-56072-9
eBook Packages: Computer ScienceComputer Science (R0)