Abstract
In this paper, we analyze a graph-theoretic property motivated by web crawling. We introduce a notion of stable cores, which is the set of web pages that are usually contained in the crawling buffer when the buffer size is smaller than the total number of web pages. We analyze the size of core in a random graph model based on the bounded Pareto power law distribution. We prove that a core of significant size exists for a large range of parameters 2 < α< 3 for the power law.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brin, S., Page, L.: The anatomy of a large-scale hypertextual Web search engine. Comput. Netw. ISDN Syst. 30, 107–117 (1998)
Bollobas, B., Riordan, O., Spencer, J., Tusnady, G.: The degree sequence of a scale-free random process. Random Structures and Algorithms 18, 279–290 (2001)
Castillo, C.: Effective Web Crawling, Ph.D. Thesis, University of Chile (2004)
Chung, F., Lu, L.: Complex Graphs and Networks. AMS (2007)
Faloutsos, C., Faloutsos, M., Faloutsos, P.: On power-law relationships of the internee topology. In: Proc. SIGCOMM (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Anderson, R., Borgs, C., Chayes, J., Hopcroft, J., Mirrokni, V., Teng, SH. (2008). On the Stability of Web Crawling and Web Search. In: Hong, SH., Nagamochi, H., Fukunaga, T. (eds) Algorithms and Computation. ISAAC 2008. Lecture Notes in Computer Science, vol 5369. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-92182-0_60
Download citation
DOI: https://doi.org/10.1007/978-3-540-92182-0_60
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-92181-3
Online ISBN: 978-3-540-92182-0
eBook Packages: Computer ScienceComputer Science (R0)