Abstract
How to give a formal description for a user’s interested topic and predict the relevance of unvisited pages to the given topic effectively is a key issue in the design of focused crawlers. However, almost all previous known focused crawlers do the Relevance Predication based on the Flat Information (RPFI) of topic only, i.e. regardless of the context between keywords or topics. In this paper, we first introduce an algorithm to map the topic described in a keyword set or a document written in natural language text to those described in hierarchical topic taxonomy. Then, we propose a novel approach to do the Relevance Predication based on the Hierarchical Context Information (RPHCI) of the taxonomy. Experiments show that the focused crawler based on RPHCI can obtain significantly higher efficiency than those based on RPFI.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Novak, B.: A Survey of Focused Web Crawling Algorithms. In: Proceedings of SIKDD 2004 at Muticonference IS, Slovennia, pp. 55–58. ACM, New York (2004)
Chau, M., Chen, H.: Comparison of Three Vertical Search Spiders. IEEE Computer 36(5), 56–62 (2003)
Hersovici, M., Jacovi, M., Maarek, Y., et al.: The Shark-Search Algorithm - An Application: Tailored Web Site Mapping. In: Proceedings of the Seventh International World Wide Web Conference, Brisbane, Australia (1998)
TREC, http://trec.nist.gov/pubs/trec12/papers/WEB.OVERVIEW.pdf
Chen, Z., Ma, J., et al.: An Improved Shark-Search Algorithm based on Link Analysis. Journal of Computational Information Systems 3(4) (2007)
Srinivasan, P., Menczer, F., Pant, G.: A General Evaluation Framework for Topical Crawlers. Information Retrieval 8(3), 417–447 (2005)
Menczer, F., Pant, G., Srinivasan, P.: Topical Web Crawlers: Evaluating Adaptive Algorithms. ACM TOIT 4(4), 378–419 (2004)
Sengor, I., Altingovde, Ulusoy, O.: Exploiting Interclass Rules for Focused Crawling. IEEE Intelligent Systems 19(6), 66–73 (2004)
Ehrig, M., Maedche, A.: Ontology-focused crawling of Web documents. In: Proc. of the 2003 ACM symposium on Applied computing, Melbourne, Florida (2003)
Open Directory Project, http://dmoz.org/
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, Z., Ma, J., Han, X., Zhang, D. (2008). An Effective Relevance Prediction Algorithm Based on Hierarchical Taxonomy for Focused Crawling. In: Li, H., Liu, T., Ma, WY., Sakai, T., Wong, KF., Zhou, G. (eds) Information Retrieval Technology. AIRS 2008. Lecture Notes in Computer Science, vol 4993. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68636-1_72
Download citation
DOI: https://doi.org/10.1007/978-3-540-68636-1_72
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68633-0
Online ISBN: 978-3-540-68636-1
eBook Packages: Computer ScienceComputer Science (R0)