Article

Information retrieval using hierarchical dirichlet processes

Author:

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 564 - 565

https://doi.org/10.1145/1008992.1009122

Published: 25 July 2004 Publication History

Get Access

Abstract

An information retrieval method is proposed using a hierarchical Dirichlet process as a prior on the parameters of a set of multinomial distributions. The resulting method naturally includes a number of features found in other popular methods. Specifically, tf.idf-like term weighting and document length normalisation are recovered. The new method is compared with Okapi BM-25 [3] and the Twenty-One model [1] on TREC data and is shown to give better performance.

References

[1]

D. Hiemstra and W. Kraaij. Twenty-one at TREC-7: Ad-hoc and cross-language track. In Text REtrieval Conference, pages 174--185, 1998.

Google Scholar

[2]

J. M. Ponte and W. B. Croft. A language modeling approach to information retrieval. In Research and Development in Information Retrieval, pages 275--281, 1998.

Digital Library

Google Scholar

[3]

S. E. Robertson, S. Walker, M. Hancock-Beaulieu, A. Gull, and M. Lau. Okapi at TREC. In Text REtrieval Conference, pages 21--30, 1992.

Google Scholar

[4]

Y. W. Teh, M. I. Jordan, M. J. Beal,and D. M. Blei. Hierarchical dirichlet processes. Technical Report 653, Department Of Statistics, UC Berkeley, 2003.

Google Scholar

Cited By

View all

Dodda RSuresh Babu A(2025)A Study of the Nonparametric Bayesian Hierarchical Dirichlet Mixture Model for Document ClusteringProceedings of the Third International Conference on Cognitive and Intelligent Computing, Volume 210.1007/978-981-97-9266-5_34(341-351)Online publication date: 26-Feb-2025
https://doi.org/10.1007/978-981-97-9266-5_34
Das SNiu YNi YMallick BPati D(2024)Blocked Gibbs Sampler for Hierarchical Dirichlet ProcessesJournal of Computational and Graphical Statistics10.1080/10618600.2024.2388543(1-11)Online publication date: 27-Sep-2024
https://doi.org/10.1080/10618600.2024.2388543
Wang ZXu SZhu L(2018)Semantic relation extraction aware of N-gram features from unstructured biomedical textJournal of Biomedical Informatics10.1016/j.jbi.2018.08.01186(59-70)Online publication date: Oct-2018
https://doi.org/10.1016/j.jbi.2018.08.011
Show More Cited By

Index Terms

Information retrieval using hierarchical dirichlet processes
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

An Online Gibbs Sampler Algorithm forźHierarchical Dirichlet Processes Prior
ECML PKDD 2016: European Conference on Machine Learning and Knowledge Discovery in Databases - Volume 9851

The hierarchical Dirichlet processes HDP is a Bayesian nonparametric model that provides a flexible mixed-membership to documents. In this paper, we develop a novel mini-batch online Gibbs sampler algorithm for the HDP which can be easily applied to ...
Using Term Location Information to Enhance Probabilistic Information Retrieval
SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval

Nouns are more important than other parts of speech in information retrieval and are more often found near the beginning or the end of sentences. In this paper, we investigate the effects of rewarding terms based on their location in sentences on ...
Variational learning of hierarchical infinite generalized Dirichlet mixture models and applications

Data clustering is a fundamental unsupervised learning task in several domains such as data mining, computer vision, information retrieval, and pattern recognition. In this paper, we propose and analyze a new clustering approach based on both ...

Comments

Information & Contributors

Information

Published In

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

July 2004

624 pages

ISBN:1581138814

DOI:10.1145/1008992

General Chair:
Mark Sanderson
University of Sheffield (UK)
,
Program Chairs:
Kalervo Järvelin
University of Tampere (Finland)
,
James Allan
University of Massachusetts (USA)
,
Peter Bruza
Distributed Systems Technology Centre (Australia)

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR04

Sponsor:

SIGIR04: The 27th ACM/SIGIR International Symposium on Information Retrieval 2004

July 25 - 29, 2004

Sheffield, United Kingdom

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
568
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dodda RSuresh Babu A(2025)A Study of the Nonparametric Bayesian Hierarchical Dirichlet Mixture Model for Document ClusteringProceedings of the Third International Conference on Cognitive and Intelligent Computing, Volume 210.1007/978-981-97-9266-5_34(341-351)Online publication date: 26-Feb-2025
https://doi.org/10.1007/978-981-97-9266-5_34
Das SNiu YNi YMallick BPati D(2024)Blocked Gibbs Sampler for Hierarchical Dirichlet ProcessesJournal of Computational and Graphical Statistics10.1080/10618600.2024.2388543(1-11)Online publication date: 27-Sep-2024
https://doi.org/10.1080/10618600.2024.2388543
Wang ZXu SZhu L(2018)Semantic relation extraction aware of N-gram features from unstructured biomedical textJournal of Biomedical Informatics10.1016/j.jbi.2018.08.01186(59-70)Online publication date: Oct-2018
https://doi.org/10.1016/j.jbi.2018.08.011
Qian JGong YZhang QHuang X(2018)Hierarchical Dirichlet Processes with Social InfluenceNatural Language Processing and Chinese Computing10.1007/978-3-319-73618-1_41(490-502)Online publication date: 5-Jan-2018
https://doi.org/10.1007/978-3-319-73618-1_41
Cummins RPaik JLv Y(2015)A Pólya Urn Document Language Model for Improved Information RetrievalACM Transactions on Information Systems10.1145/274623133:4(1-34)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1145/2746231
Dai AStorkey A(2015)The Supervised Hierarchical Dirichlet ProcessIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2014.231580237:2(243-255)Online publication date: 1-Feb-2015
https://dl.acm.org/doi/10.1109/TPAMI.2014.2315802
Bejan CHarabagiu S(2014)Unsupervised event coreference resolutionComputational Linguistics10.1162/COLI_a_0017440:2(311-347)Online publication date: 1-Jun-2014
https://dl.acm.org/doi/10.1162/COLI_a_00174
Zhang MHe TLi FPeng L(2013)Incorporating Hierarchical Dirichlet Process into Tag Topic ModelChinese Lexical Semantics10.1007/978-3-642-45185-0_39(368-377)Online publication date: 2013
https://doi.org/10.1007/978-3-642-45185-0_39
Haffari GTeh YOstendorf MCollins MNarayanan SOard D(2009)Hierarchical dirichlet trees for information retrievalProceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics10.5555/1620754.1620780(173-181)Online publication date: 31-May-2009
https://dl.acm.org/doi/10.5555/1620754.1620780
Nomoto TMooney R(2005)Bayesian learning in text summarizationProceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing10.3115/1220575.1220607(249-256)Online publication date: 6-Oct-2005
https://dl.acm.org/doi/10.3115/1220575.1220607

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

An Online Gibbs Sampler Algorithm forźHierarchical Dirichlet Processes Prior

Using Term Location Information to Enhance Probabilistic Information Retrieval

Variational learning of hierarchical infinite generalized Dirichlet mixture models and applications

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations