Article

Active feedback in ad hoc information retrieval

Authors:

ChengXiang ZhaiAuthors Info & Claims

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 59 - 66

https://doi.org/10.1145/1076034.1076047

Published: 15 August 2005 Publication History

Abstract

Information retrieval is, in general, an iterative search process, in which the user often has several interactions with a retrieval system for an information need. The retrieval system can actively probe a user with questions to clarify the information need instead of just passively responding to user queries. A basic question is thus how a retrieval system should propose questions to the user so that it can obtain maximum benefits from the feedback on these questions. In this paper, we study how a retrieval system can perform active feedback, i.e., how to choose documents for relevance feedback so that the system can learn most from the feedback information. We present a general framework for such an active feedback problem, and derive several practical algorithms as special cases. Empirical evaluation of these algorithms shows that the performance of traditional relevance feedback (presenting the top K documents) is consistently worse than that of presenting documents with more diversity. With a diversity-based selection algorithm, we obtain fewer relevant documents, however, these fewer documents have more learning benefits.

References

[1]

J. Allan. HARD track overview in TREC2003. In Proceedings of TREC 2003, 2003.

[2]

J. Carbonell and J. Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summarires. In Proceedings of SIGIR 1998, 1998.

Digital Library

[3]

D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129--145, 1996.

Digital Library

[4]

D. Harman. Relevance feedback revisited. In Proceedings of SIGIR 1998, 1992.

Digital Library

[5]

T. Jaakkola and H. Siegelmann. Active information retrieval. In Proceedings of NIPS 2001, 2001.

[6]

T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of SIGKDD 2002, 2002.

Digital Library

[7]

L. Kaufman and P. J. Rousseeuw. Finding Groups in Data: an Introduction to Cluster Analysis. John Wiley & Sons, 1990.

[8]

D. Kelly and J. Teevan. Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2):18--28, 2003.

Digital Library

[9]

J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In Proceedings of SIGIR 2001, pages 111--119, 2001.

Digital Library

[10]

D. D. Lewis. Active by accident: Relevance feedback in information retrieval. Unpublished Working Notes of 1995 AAAI Fall Symposium on Active Learning, 1995.

[11]

D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of ICML 1994, 1994.

[12]

D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In Proceedings of SIGIR 1994, pages 3--12, 1994.

Digital Library

[13]

J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145--151, 1991.

Digital Library

[14]

A. K. McCallum and K. Nigam. Employing EM in pool-based active learning for text classification. In Proceedings of ICML 1998, 1998.

Digital Library

[15]

S. E. Robertson, H. Zaragoza, and M. Taylor. Microsoft Cambridge at TREC-12: HARD track. In Proceedings of TREC 2003, 2003.

[16]

J. J. Rocchio. Relevance feedback in information retrieval. The SMART Retrieval System, pages 313--323, 1971.

[17]

N. Roy and A. McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of ICML 2001, 2001.

Digital Library

[18]

G. Salton and C. Buckley. Improving retrieval performance by retrieval feedback. Journal of the American Society for Information Science, 41(4):288--297, 1990.

[19]

G. Schohn and D. Cohn. Less is more: Active learning with support vector machine. In Proceedings of ICML 2001, pages 839--846, 2001.

Digital Library

[20]

X. Shen and C. Zhai. Active feedback--UIUC TREC2003 HARD experiments. In Proceedings of TREC 2003, 2003.

[21]

K. Sparck Jones. Search term relevance weighting given little relevance information. Journal of Documentation, 35(1):30--48, 1979.

[22]

S. Tong. Active Learning: Theory and Applications. PhD thesis, Stanford University, 2001.

Digital Library

[23]

S. Tong and D. Koller. Support vector machine active learning with applications to text classification. In Proceedings of ICML 2000, 2000.

Digital Library

[24]

Lemur Toolkit. http://www.cs.cmu.edu/~lemur.

[25]

C. Zhai. Risk Minimization and Language Modeling in Text Retrieval. PhD thesis, Carnegie Mellon University, 2002.

[26]

C. Zhai, W. W. Cohen, and J. Lafferty. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In Proceedings of SIGIR 2003, pages 10--17, 2003.

Digital Library

[27]

C. Zhai and J. Lafferty. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of CIKM 2001, pages 403--410, 2001.

Digital Library

[28]

C. Zhang and T. Chen. An active learning framework for content-based information retrieval. IEEE Transactions on Multimedia, 4:260--268, 2002.

Digital Library

[29]

Y. Zhang, W. Xu, and J. P. Callan. Exploration and exploitation in adaptive filtering based on Bayesian active learning. In Proceedings of ICML 2003, 2003.

Cited By

Althammer SZuccon GHofstätter SVerberne SHanbury A(2023)Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random SelectionProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625333(139-149)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625333
Jin QTan CChen MYan MZhang NHuang SLiu X(2022)State-of-the-Art Evidence Retriever for Precision Medicine: Algorithm Development and ValidationJMIR Medical Informatics10.2196/4074310:12(e40743)Online publication date: 15-Dec-2022
https://doi.org/10.2196/40743
Son KKim KHyun K(2022)BIGexplore: Bayesian Information Gain Framework for Information ExplorationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517729(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517729
Show More Cited By

Index Terms

Active feedback in ad hoc information retrieval
1. Information systems
  1. Information retrieval
  2. Information systems applications
    1. Data mining
      1. Clustering

Recommendations

Adaptive relevance feedback in information retrieval
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Relevance Feedback has proven very effective for improving retrieval accuracy. A difficult yet important problem in all relevance feedback methods is how to optimally balance the original query and feedback information. In the current feedback methods, ...
A split-list approach for relevance feedback in information retrieval

In this paper we present a new algorithm for relevance feedback (RF) in information retrieval. Unlike conventional RF algorithms which use the top ranked documents for feedback, our proposed algorithm is a kind of active feedback algorithm which ...
Information retrieval with concept-based pseudo-relevance feedback in MEDLINE

Although using domain specific knowledge sources for information retrieval yields more accurate results compared to pure keyword-based methods, more improvements can be achieved by considering both relations between concepts in an ontology and also their ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

August 2005

708 pages

ISBN:1595930345

DOI:10.1145/1076034

General Chairs:
Ricardo Baeza-Yates
University of Chile, Chile
,
Nivio Ziviani
Federal University of Minas Gerais, Brazil
,
Program Chairs:
Gary Marchionini
University of North Carolina, USA
,
Alistair Moffat
University of Melbourne, Australia
,
John Tait
University of Sunderland, UK

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR05

Sponsor:

SIGIR

SIGIR05: The 28th ACM/SIGIR International Symposium on Information Retrieval 2005

August 15 - 19, 2005

Salvador, Brazil

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
1,445
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Althammer SZuccon GHofstätter SVerberne SHanbury A(2023)Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random SelectionProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625333(139-149)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625333
Jin QTan CChen MYan MZhang NHuang SLiu X(2022)State-of-the-Art Evidence Retriever for Precision Medicine: Algorithm Development and ValidationJMIR Medical Informatics10.2196/4074310:12(e40743)Online publication date: 15-Dec-2022
https://doi.org/10.2196/40743
Son KKim KHyun K(2022)BIGexplore: Bayesian Information Gain Framework for Information ExplorationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517729(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517729
Kim MKim YLee EHong JBures MPark JCerny T(2022)How does the first buggy file work well for iterative IR-based bug localization?Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507034(1509-1516)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3477314.3507034
Albahem ASpina DScholer FCavedon L(2021)Component-based Analysis of Dynamic Search PerformanceACM Transactions on Information Systems10.1145/348323740:3(1-47)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.1145/3483237
Rokni SNourollahi MAlinia PMirzadeh IPedram MGhasemzadeh H(2020)TransNetACM Transactions on Design Automation of Electronic Systems10.1145/341406226:1(1-31)Online publication date: 10-Sep-2020
https://dl.acm.org/doi/10.1145/3414062
Ma LDing BDas SSwaminathan AMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Active Learning for ML Enhanced Database SystemsProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3389768(175-191)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3389768
Esna Ashari ZChaytor NCook DGhasemzadeh H(2020)Memory-Aware Active Learning in Mobile Sensing SystemsIEEE Transactions on Mobile Computing10.1109/TMC.2020.3003936(1-1)Online publication date: 2020
https://doi.org/10.1109/TMC.2020.3003936
Luo RWang X(2020)Batch Active Learning With Two-Stage SamplingIEEE Access10.1109/ACCESS.2020.29793158(46518-46528)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2979315
Ashari ZGhasemzadeh H(2019)Mindful active learningProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367243.3367354(2265-2271)Online publication date: 10-Aug-2019
https://dl.acm.org/doi/10.5555/3367243.3367354
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten