Mingfang Wu
Falk Scholer
Andrew Turpin
ABSTRACT
Question answering systems increasingly need to deal with complex information needs that require more than simple factoid answers. The evaluation of such systems is usually carried out using precision- or recall-based system performance metrics. Previous work has demonstrated that when users are shown two search result lists side-by-side, they can reliably differentiate between the qualities of the lists. We investigate the consistency between this user-based approach and system-oriented metrics in the question answering environment. Our initial results indicate that the two methodologies show a high level of disagreement.
- H. T. Dang and D. Kelly and J. Lin. Overview of the TREC-2007 Question Answering Track. In TREC 2007 Notebook, November 2007, Gaithersburg, Maryland.Google Scholar
- J. Lin and D. Demner-Fushman. Will Pyramids built of nuggets topple over? Proc HLT/EMNLP 2006, pp.383--390. Google ScholarDigital Library
- I. Ruthven and L. Azzopardi et al. Intra-assessor consistency in question answering. Proc SIGIR 2007, pp.727--728. Google ScholarDigital Library
- P. Thomas and D. Hawking. Evaluation by Comparing Result Sets in Context, Proc CIKM 2006, pp.94--101. Google ScholarDigital Library
Index Terms
- User preference choices for complex question answering
Recommendations
Human question answering performance using an interactive document retrieval system
IIIX '12: Proceedings of the 4th Information Interaction in Context SymposiumEvery day, people answer their questions by using document retrieval systems. Compared to document retrieval systems, question answering (QA) systems aim to speed the rate at which users find answers by retrieving answers rather than documents. To ...
Quality-aware collaborative question answering: methods and evaluation
WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data MiningCommunity Question Answering (QA) portals contain questions and answers contributed by hundreds of millions of users. These databases of questions and answers are of great value if they can be used directly to answer questions from any user. In this ...
Efficient top-k retrieval for user preference queries
SAC '11: Proceedings of the 2011 ACM Symposium on Applied ComputingEfficient retrieval of the most relevant (i.e. top-k) tuples is an important requirement in information systems which access large amounts of data. In general answering a top-k query request means to retrieve the k-objects which score best for an ...
Comments