Abstract
Question Answering (QA) Systems are put forward as a real alternative to Information Retrieval systems as they provide the user with a fast and comprehensible answer to his or her information need. It has been 15 years since TREC introduced the first QA track. The principal campaigns in the evaluation of Information Retrieval have been specific tracks focusing on the development and evaluation of this type of system. This study is a brief review of the TREC, CLEF and NTCIR Conferences from the QA perspective. We present a historical overview of 15 years of QA evaluation tracks using the method of systematic review. We have examined identified the different tasks or specific labs created in each QA track, the types of evaluation question used, as well as the evaluation measures used in the different competitions analyzed. Of the conferences, it is CLEF that has applied the greater variety of types of test question (factoid, definition, list, causal, yes/no, amongst others). NTCIR, held on 13 occasions, is the conference which has made use of a greater number of different evaluation measures. Accuracy, precision and recall have been the three most used evaluation measures in the three campaigns.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dwivedi, S.K., Singh, V.: Research and Reviews in Question Answering System. Procedia Technology 10, 417–424 (2013)
Peñas, A., Magnini, B., Forner, P., Sutcliffe, R., Rodrigo, A., Giampiccolo, D.: Question answering at the cross-language evaluation forum 2003—2010. Lang. Resour. Eval. 46(2), 177–217 (2012)
Mollá, D., Vicedo, J.L.: Question Answering in Restricted Domains: An Overview. Comput. Linguist. 33(1), 41–61 (2007)
Diekerma, A.R., Yilmazel, O., Liddy, E.D.: Evaluation of restricted domain Question-Answering systems Center for Natural Language Processing. Paper 3 (2004)
Sing, G.O., Ardil, C., Wong, W., Sahib, S.: Response Quality Evaluation in Heterogeneous Question Answering System: A Black-box Approach. In: Proceedings of World Academy of Science, Lisbon, vol. 9 (2005)
Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Question-answering systems as efficient sources of terminological information: an evaluation. Health Information & Libraries Journal 27, 268–276 (2010)
Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Language resources used in multi‐lingual question‐answering systems. Online Information Review 35(4), 543–557 (2011)
Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Multilingual Question-Answering System in biomedical domain on the Web: an evaluation. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 83–88. Springer, Heidelberg (2011)
Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Evaluation of Open- vs. Restricted- Domain Question Answering Systems in the Biomedical Field. Journal of Information Science 37(2), 152–162 (2011)
Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general. Revista española de Documentación Científica 36(2), e9 (2013)
Radev, D.R., Qi, H., Wu, H., Weiguo, F.: Evaluating Web-based Question Answering Systems. In: Proceedings LREC (2002)
Forner, P., Giampiccolo, D., Magnini, B., Peñas, A., Rodrigo, Á., Sutcliffe, R.: Evaluating multilingual question answering systems at CLEF. In: Proc. 7th International Conference on Language Resources and Evaluation. LREC 2010, Malta, pp. 2774–2781 (2010)
Voorhees, E.: Overview of TREC 2003. In: Proceedings of the 12th Text Retrieval Conference (2003)
Kelly, D., Sugimoto, C.R.: A systematic review of interactive information retrieval evaluation studies, 1967–2006. Journal of the American Society for Information Science and Technology 64, 745–770 (2013)
Moghadasi, S.I., Ravana, S.D.: Low-cost evaluation techniques for information retrieval systems: A review. Journal of Informetrics 7(2), 301–312 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Olvera-Lobo, MD., Gutiérrez-Artacho, J. (2015). Question Answering Track Evaluation in TREC, CLEF and NTCIR. In: Rocha, A., Correia, A., Costanzo, S., Reis, L. (eds) New Contributions in Information Systems and Technologies. Advances in Intelligent Systems and Computing, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-319-16486-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-16486-1_2
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16485-4
Online ISBN: 978-3-319-16486-1
eBook Packages: Computer ScienceComputer Science (R0)