article

Overview of the TREC 2006 ciQA task

Authors:

Diane Kelly,

Jimmy LinAuthors Info & Claims

ACM SIGIR Forum, Volume 41, Issue 1

Pages 107 - 116

https://doi.org/10.1145/1273221.1273231

Published: 01 June 2007 Publication History

Get Access

Abstract

Growing interest in interactive systems for answering complex questions lead to the development of the complex, interactive QA (ciQA) task, introduced for the first time at TREC 2006. This paper describes the rationale and design of the ciQA task and the evaluation results. Thirty complex relationship questions based on five question templates were investigated using the AQUAINT collection of newswire text. Interaction forms were the primary vehicle for defining and capturing user-system interactions. In total, six groups participated in the ciQA task and contributed ten different sets of interaction forms. There were two main findings: baseline IR techniques are competitive for complex QA and interaction, at least as defined and implemented in this evaluation, did not appear to improve performance by much.

References

[1]

J. Allan. 2005. HARD track overview in TREC 2005: High accuracy retrieval from documents. In TREC 2005.

Google Scholar

[2]

H. Dang, J. Lin, and D. Kelly. 2006. Overview of the TREC 2006 question answering track. In TREC 2006.

Google Scholar

[3]

S. Dumais and N. Belkin. 2005. The TREC interactive tracks: Putting the user into search. In Voorhees and Harman, editors, TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA.

Google Scholar

[4]

W. Hersh and P. Over. 2001. Interactivity at the Text Retrieval Conference (TREC). IP&M, 37(3):365--367.

Google Scholar

[5]

W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. 2005. TREC 2005 Genomics Track overview. In TREC 2005.

Google Scholar

[6]

J. Lin and D. Demner-Fushman. 2006. Will pyramids built of nuggets topple over? In HLT/NAACL 2006.

Digital Library

Google Scholar

[7]

A. Spink. 1997. Study of interactive feedback during mediated information retrieval. JASIS, 48(5):382--394.

Crossref

Google Scholar

[8]

E. Voorhees and H. Dang. 2005. Overview of the TREC 2005 question answering track. In TREC 2005.

Google Scholar

[9]

E. Voorhees. 2003. Overview of the TREC 2003 question answering track. In TREC 2003.

Google Scholar

Cited By

View all

Zuo CMathur KKela DSalek Faramarzi NBanerjee R(2022)Beyond belief: a cross-genre study on perception and validation of health information onlineInternational Journal of Data Science and Analytics10.1007/s41060-022-00310-713:4(299-314)Online publication date: 2-Feb-2022
https://doi.org/10.1007/s41060-022-00310-7
Chen CBunescu RMarling C(2021)A semantic parsing pipeline for context-dependent question answering over temporally structured dataNatural Language Engineering10.1017/S1351324921000292(1-25)Online publication date: 29-Oct-2021
https://doi.org/10.1017/S1351324921000292
Eickhoff CDungs STran VBaeza-Yates RLalmas MMoffat ARibeiro-Neto B(2015)An Eye-Tracking Study of Query ReformulationProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767703(13-22)Online publication date: 9-Aug-2015
https://dl.acm.org/doi/10.1145/2766462.2767703
Show More Cited By

Index Terms

Overview of the TREC 2006 ciQA task
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

The TREC 2005 robust track

The TREC robust retrieval track explores methods for improving the consistency of retrieval technology by focusing on poorly performing topics. The retrieval task in the track is a traditional ad hoc retrieval task where the evaluation methodology ...
The TREC files: the (ground) truth is out there
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Traditional tools for information retrieval (IR) evaluation, such as TREC's trec_eval, have outdated command-line interfaces with many unused features, or 'switches', accumulated over the years. They are usually seen as cumbersome applications by new IR ...
Liberal relevance criteria of TREC -: counting on negligible documents?
SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

Most test collections (like TREC and CLEF) for experimental research in information retrieval apply binary relevance assessments. This paper introduces a four-point relevance scale and reports the findings of a project in which TREC-7 and TREC-8 ...

Comments

Information & Contributors

Information

Published In

ACM SIGIR Forum Volume 41, Issue 1

June 2007

100 pages

ISSN:0163-5840

DOI:10.1145/1273221

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2007

Published in SIGIR Volume 41, Issue 1

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
223
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zuo CMathur KKela DSalek Faramarzi NBanerjee R(2022)Beyond belief: a cross-genre study on perception and validation of health information onlineInternational Journal of Data Science and Analytics10.1007/s41060-022-00310-713:4(299-314)Online publication date: 2-Feb-2022
https://doi.org/10.1007/s41060-022-00310-7
Chen CBunescu RMarling C(2021)A semantic parsing pipeline for context-dependent question answering over temporally structured dataNatural Language Engineering10.1017/S1351324921000292(1-25)Online publication date: 29-Oct-2021
https://doi.org/10.1017/S1351324921000292
Eickhoff CDungs STran VBaeza-Yates RLalmas MMoffat ARibeiro-Neto B(2015)An Eye-Tracking Study of Query ReformulationProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767703(13-22)Online publication date: 9-Aug-2015
https://dl.acm.org/doi/10.1145/2766462.2767703
Elbedweihy KWrigley SClough PCiravegna F(2015)An overview of semantic search evaluation initiativesWeb Semantics: Science, Services and Agents on the World Wide Web10.1016/j.websem.2014.10.00130:C(82-105)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.1016/j.websem.2014.10.001
Hovy ENavigli RPonzetto S(2013)Collaboratively built semi-structured content and Artificial IntelligenceArtificial Intelligence10.1016/j.artint.2012.10.002194(2-27)Online publication date: 1-Jan-2013
https://dl.acm.org/doi/10.1016/j.artint.2012.10.002
Liu SZhong YRen F(2013)Interactive Question Answering Based on FAQChinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data10.1007/978-3-642-41491-6_8(73-84)Online publication date: 2013
https://doi.org/10.1007/978-3-642-41491-6_8
Peñas AMagnini BForner PSutcliffe RRodrigo ÁGiampiccolo D(2012)Question answering at the cross-language evaluation forum 2003---2010Language Resources and Evaluation10.1007/s10579-012-9177-046:2(177-217)Online publication date: 1-Jun-2012
https://dl.acm.org/doi/10.1007/s10579-012-9177-0
Elbedweihy KWrigley SCiravegna F(2012)Evaluating semantic search query approaches with expert and casual usersProceedings of the 11th international conference on The Semantic Web - Volume Part II10.1007/978-3-642-35173-0_18(274-286)Online publication date: 11-Nov-2012
https://dl.acm.org/doi/10.1007/978-3-642-35173-0_18
Ruthven I(2011)Information Retrieval in ContextAdvanced Topics in Information Retrieval10.1007/978-3-642-20946-8_8(187-207)Online publication date: 2011
https://doi.org/10.1007/978-3-642-20946-8_8
Achananuparp PHu XHe TYang CAn YGuo L(2010)Answer diversification for complex question answering on the webProceedings of the 14th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part I10.1007/978-3-642-13657-3_41(375-382)Online publication date: 21-Jun-2010
https://dl.acm.org/doi/10.1007/978-3-642-13657-3_41
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

The TREC 2005 robust track

The TREC files: the (ground) truth is out there

Liberal relevance criteria of TREC -: counting on negligible documents?

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations