Skip to main content
SpringerLink
Log in

Question answering at the cross-language evaluation forum 2003–2010

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

The paper offers an overview of the key issues raised during the 8 years’ activity of the Multilingual Question Answering Track at the Cross Language Evaluation Forum (CLEF). The general aim of the track has been to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages, also drawing attention to a number of challenging issues for research in multilingual QA. The paper gives a brief description of how the task has evolved over the years and of the way in which the data sets have been created, presenting also a short summary of the different types of questions developed. The document collections adopted in the competitions are outlined as well, and data about participation is provided. Moreover, the main measures used to evaluate system performances are explained and an overall analysis of the results achieved is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. http://www.clef-campaign.org.

  2. http://wt.jrc.it/lt/Acquis/.

  3. http://www.europarl.europa.eu.

  4. http://nlp.uned.es/clef-qa/ave/.

  5. Currently all the material is available at http://celct.isti.cnr.it/ResPubliQA/index.php?page=Pages/pastCampaigns.php.

  6. http://celct.isti.cnr.it/ResPubliQA.

References

  • Agirre, E., Ansa, O., Arregi, X., Lopez de Lacalle, M., Otegi, A., Saralegi, Z., et al. (2009). ElhuyarIXA: Semantic relatedness and crosslingual passage retrieval. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, 30 September–2 October. Lecture notes in computer science 6241. Springer (Revised selected papers).

  • Agirre, E., Ansa, O., Arregi, X., Lopez de Lacalle, M., Otegi, A., & Saralegi, X. (2010). Document expansion for cross-lingual passage retrieval. In M. Braschler, D. Harman, & E. Pianta (Eds.), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010 Padua, Italy.

  • Amaral, C., Figueira, H., Martins, A., Mendes, A., Mendes, P., & Pinto, C. (2005). Priberam’s question answering system for Portuguese. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).

  • Amaral, C., Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., et al. (2007). Priberam’s question answering system in QA@CLEF 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).

  • Amaral, C., Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., et al. (2008). Priberam’s question answering system in QA@CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).

  • Bouma, G., Mur, J., van Noord, G., van der Plas, L., & Tiedemann, J. (2005). Question answering for Dutch using dependency relations. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).

  • Bowden, M., Olteanu, M., Suriyentrakorn, P., d’Silva, T., & Moldovan. D. (2007). Multilingual question answering through intermediate translation: LCC’s PowerAnswer at QA@CLEF 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).

  • Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., Pinto, C., et al. (2006). Priberam’s question answering system in a cross-language environment. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).

  • Clarke, C., Cormack, G., & Lynam, T. (2001). Exploiting redundancy in question answering. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR-2001). September 9–13, 2001, New Orleans, Louisiana. ACM 2001, ISBN 1-58113-331-6.

  • Correa, S., Buscaldi, D., & Rosso, P. (2009). NLEL-MAAT at CLEF-ResPubliQA. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).

  • Dornescu, I., Puscasu, G., & Orsasan, C. (2008). University of Wolverhampton at CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).

  • Ferrandez, O., Spurk, C., Kouylekov, M., Dornescu, I., Ferrandez, S., Negri, M., Izquierdo, R., Tomas, D., Orasan, C., Neumann, G., Magnini B., & Vicedo, J. L. (2011). The QALL-ME framework: A specifiable-domain multilingual question answering architecture. Web semantics: Science, services and agents on the world wide web, 9 (2), Provenance in the Semantic Web, 137–145.

  • Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., et al. (2010). Building Watson: An overview of the DeepQA project. AI Magazine, 31(3), 59–79.

    Google Scholar 

  • Forner, P., Peñas, A., Alegria, I., Forascu, C., Moreau, N., Osenova, P., et al. (2008). Overview of the CLEF 2008 multilingual question answering track. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).

  • Giampiccolo, D., Forner, P., Herrera, J., Peñas, A., Ayache, C., Forascu, C., et al. (2007). Overview of the CLEF 2007 Multilingual Question Answering Track. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).

  • Gillard, L., Sitbon, L., Blaudez, E., Bellot, P., & El-Bèze, M. (2006). The LIA at QA@CLEF-2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).

  • Gonzalo, J., Clough, P., & Vallin, A. (2005). Overview of the CLEF 2005 interactive track. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).

  • Hartrumpf, S. (2004). Question answering using sentence parsing and semantic network matching. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, September 15–17, 2004, (Revised selected papers).

  • Herrera, J., Peñas, A., & Verdejo, F. (2005). Question answering pilot task at CLEF 2004. Multilingual information access for text, speech and images. CLEF 2004. Vol. 3491 of lecture notes in computer science, pp. 581–590.

  • Hirschman, L., & Gaizauskas. R. (2001). Natural language question answering: the view from here. In natural language engineering, Vol. 7(4), December 2001, pp. 275–300.

  • Hovy, E., Gerber, L., Hermjakob, H., Lin, C., & Ravichandran, D. (2001). Toward semantics-based answer pinpointing. In Proceedings of the DARPA human language technology conference (HLT), San Diego, CA.

  • Iftene, A., Trandabat, D., Pistol, I., Moruz, A. M., Husarciuc, M., Sterpu, M., & Turliuc, C. (2009). Question answering on english and romanian languages. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. i text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).

  • Ion, R., Ştefănescu, D., Ceauşu, A., Tufiş, D., Irimia, E., & Barbu-Mititelu, V. (2009). A trainable multi-factored QA system. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).

  • Jijkoun, V., & de Rijke, M. (2007). Overview of the WiQA task at CLEF 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).

  • Jijkoun, V., Mishne, G., & de Rijke, M. (2003). The University of Amsterdam at QA@CLEF2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).

  • Jijkoun, V., Mishne, G., de Rijke, M., Schlobach, S., Ahn, D., & Muller, H. (2004). The University of Amsterdam at QA@CLEF 2004. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, September 15–17, 2004 (Revised selected papers).

  • Kelly, D., & Lin, J. (2007). Overview of the TREC 2006 ciQA task. SIGIR Forum, 41, 1.

    Article  Google Scholar 

  • Lamel, L., Rosset, S., Ayache, C., Mostefa, D., Turmo, J., Comas P. (2007). Question answering on speech transcriptions: The QAST evaluation in CLEF. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, & A. Peñas, D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).

  • Laurent, D., Séguéla, P., & Nègre, S. (2005). Cross lingual question answering using QRISTAL for CLEF 2005. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).

  • Laurent, D., Séguéla, P., & Nègre, S. (2006). Cross lingual question answer ing using QRISTAL for CLEF 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).

  • Laurent, D., Séguéla, P., & Nègre, S. (2007). Cross lingual question answering using QRISTAL for CLEF 2007. In: C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).

  • Laurent, D., Séguéla, P., & Nègre, S. (2008). Cross lingual question answering using QRISTAL for CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).

  • Lopez, V., Uren, V. S., Sabou, M., & Motta, E. (2009). Cross ontology query answering on the semantic web: An initial evaluation. In K-CAP-2009: Proceedings of the fifth international conference on knowledge capture, Redondo Beach, CA.

  • Magnini, B., Negri, M., Prevete, R., & Tanev, H. (2002). Is it the right answer? Exploiting web redundancy for answer validation In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL), Philadelphia. doi:10.3115/1073083.1073154.

  • Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Peñas, A., Peinado, V., et al. (2003). The multiple language question answering track at CLEF 2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).

  • Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., de Rijke, M., Rocha, P., Simov, K., & Sutcliffe, R. (2004). Overview of the CLEF 2004 multilingual question answering track. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, UK, September 15–17, 2004 (Revised selected papers).

  • Magnini, B., Giampiccolo, D., Forner, P., Ayache, C., Jijkoun, V., Osenova, P., et al. (2006). Overview of the CLEF 2006 multilingual question answering track. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th Workshop of the cross-language evaluation forum, CLEF 2006, Alicante, September 20–22, 2006 (Revised selected papers).

  • Magnini, B., Speranza, M., & Kumar, V. (2009). Towards interactive question answering: an ontology-based approach. In Proceedings of the IEEE international conference on semantic computing, September 22–24, 2010, Carnegie Mellon University, Pittsburgh, PA.

  • Negri, M., Tanev, H., & Magnini, B. (2003) Bridging Languages for Question Answering: DIOGENE at CLEF-2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, August 21–22, 2003 (Revised selected papers).

  • Noguera, E., Llopis, F., Ferrandez, A., & Escapa, A. (2007). Evaluation of open-domain question answering systems within a time constraint. In Advanced Information networking and applications workshops, AINAW ‘07. 21st international conference.

  • Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL), Philadelphia, July 2002.

  • Peñas, A., & Rodrigo, Á. (2011) A Simple Measure to Assess Non-response. In Proceedings of 49th annual meeting of the association for computational linguistics—human language technologies (ACL-HLT 2011), Portland, Oregon, June 19–24, 2011.

  • Peñas, A., Rodrigo, A., Sama, V., & Verdejo, F. (2006). Overview of the answer validation exercise 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).

  • Peñas, A., Rodrigo, Á., & Verdejo, F. (2007). Overview of the Answer Validation Exercise 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, V. Petras, & D. Santos, (Eds.), Advances in multilingual and multimodal information retrieval, LNCS 5152, September 2008.

  • Peñas, A., Forner, P., Sutcliffe, R., Rodrigo, Á., Forascu, C., Alegria, I., et al. (2009). Overview of ResPubliQA 2009: Question answering evaluation over European legislation. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. i text retrieval experiments, workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2. Lecture Notes in computer science 6241. Springer-Verlag, 2010. (Revised selected papers).

  • Peñas, A., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, & Mota, C. (2010). Overview of ResPubliQA 2010: Question answering evaluation over european legislation. In M. Braschler, D. Harman, and E. Pianta (Eds,), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010, Padua, Italy.

  • Pérez-Iglesias, J, Garrido, G., Rodrigo, Á, Araujo, L., & Peñas, A. (2009). Information retrieval baselines for the ResPubliQA Task. In Borri, F., Nardi, A., & Peters, C. (Eds.), Cross language evaluation forum: Working notes of CLEF 2009, Corfu, Greece, September 30–October 2.

  • Prager, J. (2006). Open-domain question-answering, Foundations and trends in information retrieval 1(2), 91–231. http:/dx.doi.org/10.1561/1500000001.

  • Prager, J., Brown, E., & Coden, A. (2000). Question-answering by predictive annotation. In Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, Athens.

  • Rodrigo, Á., Peñas, A., & Verdejo, F. (2008). Overview of the answer validation exercise 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, September 17–19, 2008 (Revised selected papers).

  • Rodrigo, A., Perez, J., Peñas, A., Garrido, G., & Araujo, L. (2009). Approaching question answering by means of paragraph validation. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer. Revised selected papers).

  • Saias, J., & Quaresma, P. (2007). The senso question answering approach to Portuguese QA@CLEF-2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, September 19–21, 2007 (Revised selected papers).

  • Saias, J., & Quaresma, P. (2008) The senso question answering system at QA@CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, September 17–19, 2008 (Revised selected papers).

  • Santos, D., & Cabral, L. M. (2009). GikiCLEF: Crosscultural issues in an international setting: Asking non-English-centered questions to Wikipedia. In Borri, F., Nardi, A., & Peters, C. (Eds.), Cross language evaluation forum: working notes of CLEF 2009, Corfu, Greece, September 30–October 2.

  • Toba, H., Sari, S., Adriani, M., & Manurung, M. (2010). Contextual approach for paragraph selection in question answering task. In M. Braschler, D. Harman, & E. Pianta (Eds.), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010, Padua, Italy.

  • Vallin, A., Magnini, B., Giampiccolo, D., Aunimo, L., Ayache, C., & Osenova, P. (2005). Overview of the CLEF 2005 multilingual question answering track. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).

  • van Rijsbergen, K. J. (1979). Information retrieval. London: Butterworth.

    Google Scholar 

  • Vicedo, J. L., Izquierdo, R., Llopis, F., & Muñoz, R. (2003). Question answering in Spanish. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).

  • Vicedo, J.L., Saiz, M., & Izquierdo, R. (2004). Does english help question answering in Spanish? In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, UK, September 15–17, 2004 (Revised selected papers).

  • Voorhees, E. M. (2000). Overview of the TREC9 question answering track. In Proceedings of the Ninth Text Retrieval Conference (TREC-9).

  • Voorhees, E. M. (2002). Overview of the TREC 2002 Question answering track. In Proceedings of the eleventh text retrieval conference (TREC-11).

  • Voorhees, E. M., Tice, D. M. (1999). The TREC-8 question answering track evaluation. In Proceedings of the eight text retrieval conference (TREC-8).

  • Webb, N., & Webber, B. (Eds.) (2009). Special issue on interactive question answering. Journal of Natural Language Engineering, 15(1), Cambridge University Press.

Download references

Acknowledgments

The QA evaluation campaigns at CLEF are a joint effort involving many institutions and numerous people who have collaborated on the creation of the data set in the various languages involved each year, and have undertaken the evaluation of the results; our appreciation and thanks (in alphabetical order) goes to: Christelle Ayache, Iñaki Alegria, Lili Aunimo, Maarten de Rijke, Gregory Erbach, Corina Forăscu, Valentin Jijkoun, Nicolas Moreau, Cristina Mota, Petya Osenova, Victor Peinado, Prokopis Prokopidis, Paulo Rocha, Bogdan Sacaleanu, Diana Santos, Kiril Simov, Erik Tjong Kim Sang, Alessandro Vallin and Felisa Verdejo. Also, support for the TrebleCLEF Coordination, within FP7 of the European Commission, Theme ICT-1-4-1 Digital Libraries and Technology Enhanced Learning (Contract 215231) must be acknowledged for its fund for ground truth creation. This work has been partially supported by the Research Network MA2VICMR (S2009/TIC-1542) and Holopedia project (TIN2010-21128-C02).

Author information

Authors and Affiliations

  1. NLP & IR Group, UNED, Madrid, Spain

    Anselmo Peñas & Álvaro Rodrigo

  2. Fondazione Bruno Kessler (FBK-irst), Trento, Italy

    Bernardo Magnini

  3. Center for the Evaluation of Language and Communication Technologies (CELCT), Trento, Italy

    Pamela Forner & Danilo Giampiccolo

  4. University of Limerick, Limerick, Ireland

    Richard Sutcliffe

Authors
  1. Anselmo Peñas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernardo Magnini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pamela Forner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard Sutcliffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Álvaro Rodrigo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Danilo Giampiccolo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pamela Forner.

Appendix: Examples of question types

Appendix: Examples of question types

Ex. 1.1: Location

Q: Where did the Purussaurus live before becoming extinct?

A: in South America’s Amazon Basin

Context (LA081694):

Its name is Purussaurus and it is the largest species of crocodile that ever lived, a massive-jawed creature that thundered through South America’s Amazon Basin 6 to 8 million years ago before its kind perished.

Ex. 1.2: Measure

Q: How long is the coastline of Santa Monica Bay?

A: 50 miles

Context (LA120994):

On Thursday, Wilson endorsed the Santa Monica Bay Restoration Project’s action plan “without condition,” calling it a vital, ambitious and broad approach to reduce pollution and restore the ecology of the bay, which stretches along 50 miles of Los Angeles County coast.

Ex. 1.3: Object

Q: What does magma consist of?

A: molten rock

Context (LA112794):

The odorless gas, more dense than air and collecting inside the floorless cabin, was welling up from a recent intrusion of magma, or molten rock, miles below the surface of this mountain that is popular with winter skiers and summer cyclists.

Ex. 1.3: Organisation

Q: What museum is directed by Henry Hopkins?

A: UCLA/Hammer Museum

Context (LA062394):

The story quotes UCLA/Hammer Museum Director Henry Hopkins as saying the Codex is among works in the Hammer collection being held in escrow as a guarantee against any financial obligation that might result from a lawsuit filed in 1990 on behalf of Joan Weiss, the niece and sole heir to the fortune of Hammer’s wife, Frances, who died in 1989.

Ex. 1.4: Person

Q: Who is John J. Famalaro accused of having killed?

A: Denise Huber

Context (LA082094):

Arizona’s governor signed extradition documents Friday that could force house painter John J. Famalaro to return to Orange County to face charges… Famalaro is accused of murdering 23-year-old Denise Huber, who vanished after her car broke down on the Corona del Mar (73) Freeway in June, 1991.

Ex. 1.5: Time

Q: When did Hernando Cortes arrive in the Aztec Empire?

A: in 1519

Context (LA112794):

When Spanish conquistador Hernando Cortes arrived in 1519, the Aztecs welcomed him, believing he was their returning god.

Ex. 1.6: Other

Q: What is another name for the “mad cow disease”?

A: bovine spongiform encephalopathy

Context (LA091194):

The government has banned foods containing intestine or thymus from calves because a new scientific study suggested that they might be contaminated with the infectious agent of bovine spongiform encephalopathy, commonly called “mad cow disease.”

Ex. 2: Definition

Q: What is Amnesty International?

A: human rights group

Context (GH951214):

The human rights group Amnesty International called Wei’s trial “a mockery of justice”.

Ex. 3: Manner

Q: How did Jimi Hendrix die?

A: drug overdose

Context (LA030994):

Hendrix Death Investigation Over: The British government said Tuesday that it would not hold a new inquest into the death 24 years ago of rock legend Jimi Hendrix, who died of a drug overdose at age 27.

Ex. 4: Temporal restriction by event

Q: Who was Uganda’s President during Rwanda’s war?

A: Yoweri Museveni

Context LA072894:

“The complicity of Ugandan President Yoweri Museveni should not be overlooked in the Rwandan crisis,” he adds.

Ex. 5: Temporal restriction by date

Q: Which city hosted the Olympic Games in 1992?

A: Barcelona

Context (LA082194):

But after the 1992 Barcelona Olympics, Henrich called Fong and announced, “Al, I’m really jazzed. I want to train for ‘96. I know I can do it.”

Ex. 6: Temporal restriction by time interval

Q: By how much did Japanese car exports fall between 1993 and 1994?

A: 18.3%

Context (LA053194):

Japan’s vehicle exports in 1993–94 fell 18.3% to 4.62 million, the second straight year of decline.

Ex. 7: Closed list question

Q: Name the three Beatles that are alive

A: Paul McCartney, George Harrison and Ringo Starr

Context (LA012994-0011):

Paul McCartney, George Harrison and Ringo Starr—the three surviving Beatles—are scheduled to reunite next month to record new music for a 10-hour video documentary, also titled “The Beatles Anthology.”

Ex. 8: Open list question

Q: What countries are members of the Gulf Cooperation Council?

A: Saudi Arabia, Kuwait, United Arab Emirates, Qatar, Oman and Bahrain

Context (LA101394-0380):

The Gulf Cooperation Council—whose members are Saudi Arabia, Kuwait, United Arab Emirates, Qatar, Oman and Bahrain—said in a formal communique that the allied military buildup “should continue until they are sure that Iraq no longer poses a threat.”

Ex. 9: Grouped questions on topic “Gulf War Syndrome”

Q1: What is Gulf War Syndrome?

A1: an inexplicable, untreatable collection of afflictions that reportedly have touched thousands who fought in the desert

Context (LA111494):

Their short, tragic lives—chronicled neatly by their mothers in family photo albums—are raising new fears that the mysterious Gulf War syndrome, an inexplicable, untreatable collection of afflictions that reportedly have touched thousands who fought in the desert, is now being passed on to the next generation.

Q2: How many people have been affected by it?

A2: 11,000

Context (LA121494):

Physicians said the 1,019 cases reflected in Tuesday’s report represent a significant sample of veterans who have sought treatment for Gulf War syndrome. Of the 697,000 who served in Operation Desert Storm, 11,000 have complained of illness. Some 8,000 are being processed.

Ex. 10: Procedure

Q: How do you find the maximum speed of a vehicle?

Paragraph (jrc31995L0001-en, para 125):

The maximum speed of the vehicle is expressed in km/h by the figure corresponding to the closest whole number to the arithmetical mean of the values for the speeds measured during the two consecutive tests, which must not diverge by more than 3%. When this arithmetical mean lies exactly between two whole members it is rounded up to the next highest number.

Ex. 11: Purpose

Q: What is the aim of the Kyoto Protocol in relation to greenhouse gas emissions for 2008-2012?

Paragraph (jrc32001Y0203_02-en, para 168):

The Kyoto Protocol, signed by the Member States and by the Community, provides that the parties undertake to limit or reduce greenhouse gas emissions during the period 2008-2012. For the Community as a whole, the target is to reduce greenhouse gas emissions by 8% of their 1990 level.

Ex. 12: Reason

Q: Why are court decisions in Kazakhstan not made public?

Paragraph (jrc21994A1231_52-en, para 1330):

The process of establishing enquiry points has begun. As far as the judicial decisions and administrative rulings are concerned they are not published in Kazakhstan (except for some decisions made by the Supreme Court), because they are not considered to be sources of law. To change the existing practice will require a long transitional period.

Ex. 13: Other

Q: During a vehicle driver’s rest period, are they entitled to a bunk?

Paragraph (jrc32006R0561-en, para 139):

1. By way of derogation from Article 8, where a driver accompanies a vehicle which is transported by ferry or train, and takes a regular daily rest period, that period may be interrupted not more than twice by other activities not exceeding 1 h in total. During that regular daily rest period the driver shall have access to a bunk or couchette.

Ex. 14: Opinion

Q: What did the Council think about the terrorist attacks on London?

Paragraph (jrc32006L0024-en, para 20):

On 13 July 2005, the Council reaffirmed in its declaration condemning the terrorist attacks on London the need to adopt common measures on the retention of telecommunications data as soon as possible.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Peñas, A., Magnini, B., Forner, P. et al. Question answering at the cross-language evaluation forum 2003–2010. Lang Resources & Evaluation 46, 177–217 (2012). https://doi.org/10.1007/s10579-012-9177-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-012-9177-0

Keywords

Search

Navigation