Abstract
Topic-driven Web Crawler or focused crawler is the key tool of on-line web information library. It’s a challenging issue that how to achieve good performance efficiently with limited time and space resources. This paper proposes a focused web crawler wHunter that implements incremental and multi-strategy learning by taking the advantages of both SVM (support vector machines) and naïve Bayes. On the one hand, the initial performance is guaranteed via SVM classifier; on the other hand, when enough web pages are obtained, the classifier is switched to naïve Bayes so that on-line incremental learning is achieved. Experimental results show that our proposed algorithm is efficient and easy to implement.
This paper has been supported by China NSF project (No.60221120145).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Chakrabarti, S., Berg, M.V.d., Dom, B.: Focused crawling: a new approach to topic-specific web resource discovery. In: Proceedings of the 8th International WWW Conference (1999)
Bra, P.D., Houben, G., Kornatzky, Y.: Information retrieval in distributed hypertexts: Making client-based searching feasible. In: Proceedings of the 1st International WWW conference (1994)
Hersovici, M., Jacovi, M., Maarek, Y.S., Pelleg, D., Shtalhaim, M., Ur, S.: The shark search algorithm-an application: Tailored web site mapping. In: Proceedings of the 7th International WWW Conference (1998)
Cho, J., Garcia-Molina, H., Page, L.: Efficient crawling through url ordering. In: Proceedings of the 7th International WWW Conference (1998)
Rennie, J., McCallum, A.K.: Using reinforcement learning to spider the web efficiently. In: Proceedings of the 16th International Conference on Machine Learning (1999)
Menczer, F., Belew, R.K.: Adaptive retrieval agents: Internalizing local context and scaling up to the web. Machine Learning 39, 203–242 (2000)
Diligenti, M., Coetzee, F.M., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: Proceedings of the 26th International Conference on Very Large Databases (2000)
Aggarwal, C.C., Al-Garawi, F., Yu, P.S.: Intelligent crawling on the world wide web with arbitrary predicates. In: Proceedings of the10th International WWW Conference (2001)
Aggarwal, C.C., Al-Garawi, F., Yu, P.S.: On the design of a learning crawler for topical resource discovery. ACM Transactions on Information Systems 19, 286–309 (2001)
Chakrabarti, S., Pinera, K., Subramanyam, M.: Accelerated focused crawling through online relevance feedback. In: Proceedings of the 11th International WWW Conference (2002)
Giraud-Carrier, C.G.: A note on the utility of incremental learning. AI Communications 13, 215–224 (2000)
Bartlett, P., Shawe-Taylor, J.: Generalization performance of support vector machines and other pattern classifiers. In: Advances in kernel methods: support vector learning. MIT Press, Cambridge (1999)
Decoste, D., Schoelkopf, B.: Training Invariant Support Vector machines. Machine Learning 46, 161–190 (2002)
Vapnik, V.N.: An overview of statistical learning theory. IEEE Transactions On Neural Networks 10(5), 988–999 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Huang, Y., Ye, Y. (2004). wHunter: A Focused Web Crawler – A Tool for Digital Library. In: Chen, Z., Chen, H., Miao, Q., Fu, Y., Fox, E., Lim, Ep. (eds) Digital Libraries: International Collaboration and Cross-Fertilization. ICADL 2004. Lecture Notes in Computer Science, vol 3334. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30544-6_59
Download citation
DOI: https://doi.org/10.1007/978-3-540-30544-6_59
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24030-3
Online ISBN: 978-3-540-30544-6
eBook Packages: Computer ScienceComputer Science (R0)