Dynamics of the Chilean Web structure

Abstract

In this paper we present a large scale study on the evolution of the Web structure of the Chilean domain (.cl) from 2000 to 2004, focusing on the Web site transitions in the structure. This is the study of the largest time span and the most detailed of its kind. Our results show that there are many stable Web sites, but also a majority of chaotic changes. We also present the first known results on the death behavior of Web sites.

Introduction

The Web is highly dynamic and not too much is known about its evolution. There has been some work on page evolution, obtaining models that predict when a page will change, but this differs a lot from site to site. There are also generative models for Web growth, but they usually do not include Web death (an exception is [5]).

In this study we focus on the Web site graph or host-graph. Web sites are better study subjects than Web pages for many reasons. First, a Web site most of the time is a logical information unit, this being less true for pages. Second, the main events on the evolution of the Web are related to sites. In fact, new Web sites appear and others disappear, but little is known about how this happens. Third, most external links in a site are to home pages, so the Web structure of sites is the glue of the Web connectivity. Fourth, most sites are strongly connected (it is enough to have a link to the home page in every page). Otherwise, a Web site would have pages in more than one component of the structure, which does not make any sense as a Web site should be atomic with respect to the overall structure (see similar and additional arguments in [6]).

The only paper that focuses in the dynamics of the host-graph is [6], but it does not study the structure of the host-graph. In [3] we presented the evolution of the structure composition of the Chilean Web at the site and domain level, based on data gathered from a search engine targeted to this country’s Internet domain, TodoCL.cl, between the years 2000 and 2002. We extended our results and their analysis to 2003 in [4]. In this paper we include data of 2004, extending our previous results and visualizations. We focus not only on macro statistics, but also on the transitions of Web sites among different structural components. That is, we try to answer the following question: are the size changes in the Web structural components due to a small number of sites going from one component to another in one direction or to a larger number of sites that go in both directions? Our results show that for some Web components the first is true, while for others the second is true.

We define the Chilean Web as all .cl sites, which in practice represent more than 98% of the sites (other non .cl sites hosted in Chile are estimated to number less than 1000). The first year the crawl started from an initial sample of sites, but subsequent years it started with all .cl domains thanks to NIC Chile (www.nic.cl). Hence, the number of unconnected sites was low the first year. Also, the last three crawls contain more dynamic pages, which in general do not change the Web structure. In addition, the last two crawls, although larger in pages compared to 2002, may not reflect an actual growth in the Chilean Web as the number of sites did not increase that much. Table 1 shows the data gathered for our study. Although our results depend on our crawling policies, we have used always the same crawler, changing only the seed URLs. Obviously, each year our seed set is larger.

Our results present how the structure evolves, how sites migrate from one component to another component, and where sites appear and disappear in the structure. The changes are dramatic, showing more chaos than order, and we elaborate on this in the conclusions. This is a first step to measure and follow the evolution of the structure of a part of the Web, as well as try to understand the process behind the changes. To the best of our knowledge there are no other studies on Web structure composition as detailed as ours, both in results and time span. Most statistical studies deal with global attributes such as language or size.

In Section 2 we review the results on the structure of the Web and the problems faced to obtain it. Section 3 shows the evolution of this structure, and Section 4 analyzes the migrations of Web sites in the structure in relation to the expected typical life cycle of a Web site. In Section 5 we analyze the dynamics of the size of Web sites. The last section contains our concluding remarks.