Abstract
The reliability of the topics within the Cross Language Evaluation Forum (CLEF) needs to be validated constantly to justify the efforts for experiments within CLEF and to demonstrate the reliability of the results as far as possible. The analysis presented in this paper is concerned with several aspects. Continuing and expanding a study from 2002, we investigate the difficulty of topics and the correlation between the retrieval quality for topics and the occurrence of proper names.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Sparck Jones, K.: Reflections on TREC. Information Processing & Management 31(3), 291–314 (1995)
Kluck, M., Womser-Hacker, C.: Inside the Evaluation Process of the Cross-Language Evaluation Forum (CLEF): Issues of Multilingual Topic Creation and Multilingual Relevance Assessment. In: Rodríguez, M.G., Araujo, C.P.S. (eds.) Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002. Las Palmas de Gran Canaria, May 29-31, 2002, pp. 573–576. ELRA, Paris (2002)
Womser-Hacker, C.: Multilingual Topic Generation within the CLEF 2001 Experiments. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 389–393. Springer, Heidelberg (2002)
Zobel, J.: How Reliable are the Results of Large-Scale Information Retrieval Experiments? In: Proceedings of the Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR 1998), Melbourne, pp. 307–314 (1998)
Voorhees, E., Buckley, C.: The Effect of Topic Set Size on Retrieval Experiment Error. In: Proceedings of the Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR 2002), Tampere, Finland, pp. 316–323 (2002)
Soboroff, I., Nicholas, C., Cahan, P.: Ranking Retrieval Systems without Relevance Judgments. In: Proceedings of the Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR 2001), New Orleans, pp. 66–73 (2001)
Voorhees, E.: Variations in relevance judgments and the measurement of retrieval effectiveness. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998), Melbourne, pp. 315–223 (1998)
Voorhees, E., Harman, D.: Overview of the Sixth Text REtrieval Conference. In: Voorhees, E., Harman, D. (eds.) The Sixth Text REtrieval Conference (TREC-6). National Institute of Standards and Technology, Gaithersburg, Maryland, NIST Special Publication (1997), http://trec.nist.gov/pubs/
Eguchi, K., Kuriyama, K., Kando, N.: Sensitivity of IR Systems Evaluation to Topic Difficulty. In: Rodríguez, M.G., Araujo, C.P.S. (eds.) Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, Las Palmas de Gran Canaria, May 29-31, 2002, pp. 585–589. ELRA, Paris (2002)
Mandl, T., Womser-Hacker, C.: Linguistic and Statistical Analysis of the CLEF Topics. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 505–511. Springer, Heidelberg (2003)
Lempel, R., Moran, S.: Predictive Caching and Prefetching of Query Results in Search Engines. In: Proceedings of the Twelfth International World Wide Web Conference (WWW 2003), Budapest, pp. 19–28. ACM Press, New York (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mandl, T., Womser-Hacker, C. (2004). Analysis of the Reliability of the Multilingual Topic Set for the Cross Language Evaluation Forum. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds) Comparative Evaluation of Multilingual Information Access Systems. CLEF 2003. Lecture Notes in Computer Science, vol 3237. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30222-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-30222-3_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24017-4
Online ISBN: 978-3-540-30222-3
eBook Packages: Springer Book Archive