A new statistical approach to estimate global file populations from local observations in the eDonkey P2P file sharing system

Abstract

In this paper, we propose a new statistical approach, also known in biology under the name capture–recapture methods in order to estimate global population statistics from local observations. Evaluating population sizes in P2P systems has received much attention lately as these may be useful to set system parameters, to derive other system statistics, or to predict system performance. As these systems are very large, encompassing several millions of users and since they are highly distributed estimating population sizes is a challenging task. More precisely, we are interested in estimating the number of file replicas in the system, i.e., the size of the population of users possessing given files. To this end, we propose a capture–recapture method which is both computationally efficient and accurate. The method proposed allows deriving global population statistics from local and time-limited observations. We apply the method on a measurement data set of several days on a residential network. We compare the results obtained from direct counting procedures with those derived with the proposed methodology.

Appendix (Proof of proposition 4)

Proof

Denote M = M _S . We prove that the ratio $\frac{r(n)}{r(n-1)}$ is larger than one below a threshold n ₀ and smaller beyond. It is sufficient to prove that for $x \in [0,\frac{1}{M}[$

$$ f(x)=\ln\left(\frac{1}{1-x M}\prod\limits_{i=0}^{S-1}(1-x C_i )\right) $$

is negative in a neighborhood of 0 and changes sign only once. Note that f(0) = 0. Also:

$$ f'(x)=\frac{M}{1-x M}-\sum\limits_{i=0}^{S-1}\frac{C_i}{1-x C_i }. $$

We have $f'(0)=M-\sum_{i=0}^{S-1}C_i<0$ as the number of different peers obtained in the end, M, must be smaller than the sum of peers obtained in each sample set, $\sum_{i=0}^{S-1}C_i$. This proves that f(x) < 0 on some interval ]0,ε[. As $\lim_{x\rightarrow {\frac{1}{M}}} f(x)=\infty$, f(x) changes sign at least once. We now prove that f(x) is convex at any point where its derivative is positive or null. The derivative must be positive or null when f(x) becomes positive for the first time. Thus, its derivative will stay positive from then on which will finish the proof. To prove convexity at points of positive derivative, we write:

$$ f"(x)=\left(\frac{M}{1-x M}\right)^2-\sum_{i=0}^{S-1}\left(\frac{C_i}{1-x C_i }\right)^2. $$

Let $a=\frac{M}{1-x M}>0$ and $b_i=\frac{C_i}{1-x C_i }>0$. Then:

$$\begin{array}{rll} f'(x)>0 &\Rightarrow& a>\sum_{i=0}^{S-1}b_i\\ &\Rightarrow& a^2>(\sum_{i=0}^{S-1}b_i)^2 >\sum_{i=0}^{S-1}b_i^2 \\ &\Rightarrow& f"(x)>0. \end{array} $$

Thus f(.) changes sign only once. □