Gaurav Baruah
Mark D. Smucker
ABSTRACT
Nugget-based evaluations, such as those deployed in the TREC Temporal Summarization and Question Answering tracks, require human assessors to determine whether a nugget is present in a given piece of text. This process, known as nugget annotation, is labor-intensive. In this paper, we present two active learning techniques that prioritize the sequence in which candidate nugget/sentence pairs are presented to an assessor, based on the likelihood that the sentence contains a nugget. Our approach builds on the recognition that nugget annotation is similar to high-recall retrieval, and we adapt proven existing solutions. Simulation experiments with four existing TREC test collections show that our techniques yield far more matches for a given level of effort than baselines that are typically deployed in previous nugget-based evaluations.
- J. Allan. HARD Track Overview in TREC 2004 High Accuracy Retrieval fromDocuments. TREC, 2004.Google Scholar
Cross Ref
- J. A. Aslam, M. Ekstrand-Abueg, V. Pavlu, F. Diaz, and T. Sakai. TREC 2013 Temporal Summarization. TREC, 2013.Google Scholar
- J. A. Aslam, M. Ekstrand-Abueg, V. Pavlu, F. Diaz, and T. Sakai. TREC 2014 Temporal Summarization. TREC, 2014.Google Scholar
- L. Azzopardi and G. Zuccon. Building and Using Models of Information Seeking Search and Retrieval: Full Day Tutorial. SIGIR, 2015. Google ScholarDigital Library
- G. Baruah, A. Roegiest, and M. D. Smucker. The Effect of Expanding Relevance Judgements with Duplicates. SIGIR, 2014. Google ScholarDigital Library
- C. L. A. Clarke and M. D. Smucker. Time Well Spent. IIiX, 2014. Google ScholarDigital Library
- G. Cormack and M. Grossman. Engineering Quality and Reliability in Technology-Assisted Review. SIGIR, 2016. Google ScholarDigital Library
- G. V. Cormack and M. R. Grossman. Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery. SIGIR, 2014. Google ScholarDigital Library
- G. V. Cormack and M. R. Grossman. Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review. CoRR, abs/1504.06868, 2015.Google Scholar
- M. Ekstrand-Abueg. Personal Communication. 2014Google Scholar
- D. Harman. Information Retrieval Evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services, 3(2), 2011. Google ScholarDigital Library
- J. He, M. Bron, A. de Vries, L. Azzopardi, and M. de Rijke. Untangling Result List Refinement and Ranking Quality: A Framework for Evaluation and Prediction. SIGIR, 2015. Google ScholarDigital Library
- J. Lin and D. Demner-Fushman. Automatically Evaluating Answers to Definition Questions. NAACL-HLT, 2005. Google ScholarDigital Library
- G. Marton and A. Radul. Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements. NAACL-HLT, 2006. Google ScholarDigital Library
- V. Pavlu, S. Rajput, P. B. Golbus, and J. A. Aslam. IR System Evaluation using Nugget-based Test Collections. WSDM, 2012. Google ScholarDigital Library
- S. Rajput, M. Ekstrand-Abueg, V. Pavlu, and J. A. Aslam. Constructing Test Collections by Inferring Document Relevance via Extracted Relevant Information. CIKM, 2012. Google ScholarDigital Library
- A. Roegiest, G. Cormack, M. Grossman, and C. Clarke. TREC 2015 Total Recall Track Overview. TREC, 2015.Google Scholar
- E. M. Voorhees. Overview of the TREC 2004 Question Answering Track. TREC, 2004.Google Scholar
- E. M. Voorhees. Overview of the TREC 2005 Question Answering Track. TREC, 2005.Google Scholar
- E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval, MIT press Cambridge, 2005. Google ScholarDigital Library
Index Terms
- Optimizing Nugget Annotations with Active Learning
Recommendations
A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge ManagementThere is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task ...
Human question answering performance using an interactive document retrieval system
IIIX '12: Proceedings of the 4th Information Interaction in Context SymposiumEvery day, people answer their questions by using document retrieval systems. Compared to document retrieval systems, question answering (QA) systems aim to speed the rate at which users find answers by retrieving answers rather than documents. To ...
Opinion retrieval from blogs
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge managementOpinion retrieval is a document retrieval process, which requires documents to be retrieved and ranked according to their opinions about a query topic. A relevant document must satisfy two criteria: relevant to the query topic, and contains opinions ...
Comments