Abstract
This paper describes the first round of ResPubliQA, a Question Answering (QA) evaluation task over European legislation, proposed at the Cross Language Evaluation Forum (CLEF) 2009. The exercise consists of extracting a relevant paragraph of text that satisfies completely the information need expressed by a natural language question. The general goals of this exercise are (i) to study if the current QA technologies tuned for newswire collections and Wikipedia can be adapted to a new domain (law in this case); (ii) to move to a more realistic scenario, considering people close to law as users, and paragraphs as system output; (iii) to compare current QA technologies with pure Information Retrieval (IR) approaches; and (iv) to introduce in QA systems the Answer Validation technologies developed in the past three years. The paper describes the task in more detail, presenting the different types of questions, the methodology for the creation of the test sets and the new evaluation measure, and analyzing the results obtained by systems and the more successful approaches. Eleven groups participated with 28 runs. In addition, we evaluated 16 baseline runs (2 per language) based only in pure IR approach, for comparison purposes. Considering accuracy, scores were generally higher than in previous QA campaigns.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Agirre, E., Ansa, O., Arregi, X., de Lacalle, M.L., Otegi, A., Saralegi, X., Zaragoza, H.: Elhuyar-IXA: Semantic Relatedness and Cross-lingual Passage Retrieval. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece.A, September 30 -October 2 (2009)
Bharadwaj, R., Ganesh, S., Varma, V.: A Naïve Approach for Monolingual Question Answering. In: Working Notes for the CLEF 2009, Workshop, Corfu, Greece, September 30 - October 2, (2009)
Correa, S., Buscaldi, D., Rosso, P.: NLEL-MAAT at CLEF-ResPubliQA. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 -October 2 (2009)
Gloeckner, I., Pelzer, B.: The LogAnswer Project at CLEF. In: Working Notes for the CLEF 2009, Workshop, Corfu, Greece, September 30 - October 2 (2009)
Iftene, A., Trandabăţ1, D., Pistol, I., Moruz1, A.-M., Husarciuc1, M., Sterpu, M., Turliuc, C.: Question Answering on English and Romanian Languages. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Ion, R., Ştefănescu, D., Ceauşu, A., Tufiş, D., Irimia, E., Barbu-Mititelu, V.: A Trainable Multi-factored QA System. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Moriceau, V., Tannier, X.: FIDJI in ResPubliQA 2009. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Peñas, A., Rodrigo, Á., Verdejo, M.F.: Overview of the Answer Validation Exercise 2007. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 237–248. Springer, Heidelberg (2008)
Peñas, A., Rodrigo, Á., Sama, V., Verdejo, M.F.: Overview of the Answer Validation Exercise 2006. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 257–264. Springer, Heidelberg (2007)
Pérez, J., Garrido, G., Rodrigo, Á., Araujo, L., Peñas, A.: Information Retrieval Baselines for the ResPubliQA Task. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: SIGIR 1994: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 232–241 (1994)
Rodrigo, Á., Peñas, A., Verdejo, F.: Overview of the Answer Validation Exercise 2008. In: Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.) Evaluating Systems for Multilingual and Multimodal Information Access. LNCS, vol. 5706, pp. 296–313. Springer, Heidelberg (2009)
Rodrigo, Á., Pérez, J., Peñas, A., Garrido, G., Araujo, L.: Approaching Question Answering by means of Paragraph Validation. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Tomlinson, S., Oard, D.W., Baron, J.R., Thompson, P.: Overview of the TREC 2007 Legal Track. In: Proceedings of The Sixteenth Text REtrieval Conference, TREC 2007, Gaithersburg, Maryland, USA, November 5-9 (2007)
Vicente-Díez, M.T., de Pablo-Sánchez, C., Martínez, P., Schneider, J.M., Salazar, M.G.: Are Passages Enough? The MIRACLE Team Participation at QA@CLEF2009. In: Working Notes for the CLEF 2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Peñas, A. et al. (2010). Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation. In: Peters, C., et al. Multilingual Information Access Evaluation I. Text Retrieval Experiments. CLEF 2009. Lecture Notes in Computer Science, vol 6241. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15754-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-15754-7_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15753-0
Online ISBN: 978-3-642-15754-7
eBook Packages: Computer ScienceComputer Science (R0)