A CPM-Based Change Detection Test for Big Data

Abstract

Big data analytics nowadays represent one of the most relevant and promising research activities in the field of Big Data. Tools and solutions designed for such purpose are meant to analyse very large sets ot data to extract relevant/valuable information. In this path, this paper addresses the problem of sequentially analysing big streams of data inspecting for changes. This problem that has been extensively studied for scalar or multivariate datastreams, has been mostly left unattended in the Big Data scenario. More specifically, the aim of this paper is to introduce a change detection test able to detect changes in datastreams characterized by very-large dimensions (up to 1000). The proposed test, based on a change-point method, is non parameteric (in the sense that it does not require any apriori information about the system under inspection or the possible changes) and is designed to detect changes in the mean vector of the datastreams. The effectiveness and the efficiency of the proposed change detection test has been tested on both synthetic and real datasets.

Appendix

As described in Sect. 4, thresholds $h_{p,\gamma ,\alpha }$s have been computed through simulations since an analytical derivation of the test statistic in (4) is hard to obtain. Thresholds have been computed as follows: for each value of p and $\alpha$ and $\gamma$, we simulated 10,000 experiments in which we randomly generated a p-variate normal distribution with random mean vector and covariance matrix. The threshold $h_{p,\gamma ,\alpha }$ is set to guarantee that the empirical probability of having a false positive detection by the proposed CDT on a fixed data-sequence whose length is $2 \gamma p$ is equal to the confidence parameter $\alpha$. Computed thresholds for different values of p and $\alpha$ and $\gamma =1.5$ are shown in Table 3. Further values of $h_{p,\gamma ,\alpha }$ for different configurations of p and $\alpha$ and $\gamma$ can be found at the following url: http://roveri.faculty.polimi.it/software-and-datasets/.

Table 3. Thresholds $h_{p,\gamma ,\alpha }$ for different values of p and $\alpha$ and $\gamma =1.5$.

To further clarify the effects of $h_{p,\gamma ,\alpha }$ in the sequential scenario, in Table 4 we detail the empirically estimated ARL for $\alpha =0.1$, $\gamma =1.5$ and p ranging from 100 to 1000.

Table 4. ARL for different values of p, $\alpha =0.1$ and $\gamma =1.5$.