Abstract
Distributed Web crawlers have recently received more and more attention from researchers. Full decentralized crawler without a centralized managing server seems to be an interesting architectural paradigm for realizing large scale information collecting systems for its scalability, failure resilience and increased autonomy of nodes. This paper provides a novel full distributed Web crawler system which is based on structured network, and a distributed crawling model is developed and applied in it which improves the performance of the system. Some important issues such as assignment of tasks, solution of scalability have been discussed. Finally, an experimental study is used to verify the advantages of system, and the results are comparatively satisfying.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Yan, H., Wang, J., Li, X., Guo, L.: Architectural Design and Evaluation of an Efficient Web-crawling System. Journal of System and Software 60(3), 185–193 (2002)
Shkapenyuk, V., Suel, T.: Design and Implementation of a High-Performance Distributed Web Crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE 2002), pp. 357–368 (2002)
Hafri, Y., Djeraba, C.: High performance crawling system. In: Proceedings of the 6th ACM SIGMM international workshop on multimedia information retrieval, pp. 299–306. ACM Press, New York (2004)
Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: A scalable fully distributed Web crawler. Software: Practice & Experience 34(8), 711–726 (2004)
Loo, B.T., Cooper, O., Krishnamurthy, S.: Distributed Web Crawling over DHTs. Tech. Rep. UCB//CSD-04-1332, UC Berkeley, Computer Science Division (February 2004)
Singh, A., Srivatsa, M., Liu, L., Miller, T.: Apoidea: A Decentralized Peer-to-Peer Architecture for Crawling the World Wide Web. In: The Proceedings of the SIGIR workshop on distributed information retrieval (August 2003)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhu, K., Xu, Z., Wang, X., Zhao, Y. (2008). A Full Distributed Web Crawler Based on Structured Network. In: Li, H., Liu, T., Ma, WY., Sakai, T., Wong, KF., Zhou, G. (eds) Information Retrieval Technology. AIRS 2008. Lecture Notes in Computer Science, vol 4993. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68636-1_51
Download citation
DOI: https://doi.org/10.1007/978-3-540-68636-1_51
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68633-0
Online ISBN: 978-3-540-68636-1
eBook Packages: Computer ScienceComputer Science (R0)