SharkFin: Spatio-temporal mining of software adoption and penetration

Abstract

How does malware propagate? Does it form spikes over time? Does it resemble the propagation pattern of benign files, such as software patches? Does it spread uniformly over countries? How long does it take for a URL that distributes malware to be detected and shut down? In this work, we answer these questions by analyzing patterns from 22 million malicious (and benign) files, found on 1.6 million hosts worldwide during the month of June 2011. We conduct this study using the WINE database available at Symantec Research Labs. Additionally, we explore the research questions raised by sampling on such large databases of executables; the importance of studying the implications of sampling is twofold: First, sampling is a means of reducing the size of the database hence making it more accessible to researchers; second, because every such data collection can be perceived as a sample of the real world. We discover the SharkFin temporal propagation pattern of executable files, the GeoSplit pattern in the geographical spread of machines that report executables to Symantec’s servers, the Periodic Power Law (Ppl) distribution of the lifetime of URLs, and we show how to efficiently extrapolate crucial properties of the data from a small sample. We further investigate the propagation pattern of benign and malicious executables, unveiling latent structures in the way these files spread. To the best of our knowledge, our work represents the largest study of propagation patterns of executables.

Appendix

The SharkFin model

The SharkFin model of executable propagation is a generalization of the SpikeM model (Matsubara et al. 2012) for the spreading of memes through blogs. We briefly describe the model in Matsubara et al. (2012), adapting to the task at hand

The model assumes a total number of $N$ machines that can be infected. Let $U(n)$ be the number of machines that are not infected at time $n;$ $I(n)$ be the count of machines that got infected up to time $n-1;$ and $\Delta I(n)$ be count of machines infected exactly at time $n.$ Then, $U(n+1)=U(n) - \Delta I(n+1)$ with initial conditions $\Delta I(0) = 0$ and $U(0) = N$.

Additionally, we let $\beta$ as the strength of that executable file. We assume that the infectiveness of a file on a machine drops as a specific power law based on the elapsed time since the file infected that machine (say $\tau$) i.e. $f(\tau ) = \beta \tau ^{-1.5}.$ Finally, we also have to consider one more parameter for our model: the "external shock”, or in other words, the first appearance of a file: let $n_b$ the time that this initial burst appeared, and let $S(n_b)$ be the size of the shock (count of infected machines).

Finally, to account for periodicity, we define a periodic function $p(n)$ with three parameters: $P_a,$ as the strength of the periodicity, $P_p$ as the period and $P_s$ as the phase shift.

Putting it all together, our SharkFin model is

$$\begin{aligned} \Delta I(n+1) = p(n+1) \left( U(n) \sum _{t = n_b}^n \left( \Delta I(t) + S(t) \right) f(n+1 - t) + \epsilon \right) \end{aligned}$$

where $p(n) = 1 - \frac{1}{2}P_a \left( \sin \left( \frac{2\pi }{P_p} \left( n+P_s\right) \right) \right) ,$ and $\epsilon$ models external noise.

No-sampling version: If $X(n), n=1\dots T$ is the sequence of file occurrences we want to model as a SharkFin spike, we want to minimize the following:

$$\begin{aligned} \min _{{ \varvec{\theta }}} \displaystyle {\sum _{n=1}^{T} \left( X(n) - \Delta I(n) \right) ^2 } \end{aligned}$$

where ${ \varvec{\theta }} = \left[\begin{array}{*{20}l} N&\beta&S_b&P_a&P_s \end{array}\right]^T$ is the vector of model parameters.

With sampling: If we are dealing with a sample of file occurrences, with sampling rate $s,$ then we solve the problem:

$$\begin{aligned} \min _{{ \varvec{\theta }}} \displaystyle {\sum _{n=1}^{T} \left( sX(n) - \Delta I(n) \right) ^2 } \end{aligned}$$

In both cases, we use Levenberg–Marquardt (Levenberg 1944) to solve for the parameters of our SharkFin model.