Unsupervised interaction-preserving discretization of multivariate data

Abstract

Discretization is the transformation of continuous data into discrete bins. It is an important and general pre-processing technique, and a critical element of many data mining and data management tasks. The general goal is to obtain data that retains as much information in the continuous original as possible. In general, but in particular for exploratory tasks, a key open question is how to discretize multivariate data such that significant associations and patterns are preserved. That is exactly the problem we study in this paper. We propose IPD, an information-theoretic method for unsupervised discretization that focuses on preserving multivariate interactions. To this end, when discretizing a dimension, we consider the distribution of the data over all other dimensions. In particular, our method examines consecutive multivariate regions and combines them if (a) their multivariate data distributions are statistically similar, and (b) this merge reduces the MDL encoding cost. To assess the similarities, we propose $ID$, a novel interaction distance that does not require assuming a distribution and permits computation in closed form. We give an efficient algorithm for finding the optimal bin merge, as well as a fast well-performing heuristic. Empirical evaluation through pattern-based compression, outlier mining, and classification shows that by preserving interactions we consistently outperform the state of the art in both quality and speed.

Appendix

1.1 Proof of Theorem 2

Proof

(Theorem 2) Let $H(\mathbf {A}) = P(\mathbf {A}) - R(\mathbf {A})$ and $G(\mathbf {A}) = R(\mathbf {A}) - Q(\mathbf {A})$. The inequality becomes

$$\begin{aligned} \sqrt{\mathop \int \limits _{\varOmega } H^2(\mathbf {a}) d\mathbf {a}} + \sqrt{\mathop \int \limits _{\varOmega } G^2(\mathbf {a}) d\mathbf {a}} \ge \sqrt{\mathop \int \limits _{\varOmega } \left( H(\mathbf {a}) + G(\mathbf {a})\right) ^2 d\mathbf {a}} , \end{aligned}$$

(9)

which in turn is equivalent to

$$\begin{aligned} \sqrt{\mathop \int \limits _{\varOmega } H^2(\mathbf {a}) d\mathbf {a}\cdot \mathop \int \limits _{\varOmega } G^2(\mathbf {a}) d\mathbf {a}} \ge \mathop \int \limits _{\varOmega } H(\mathbf {a}) G(\mathbf {a}) d\mathbf {a}, \end{aligned}$$

(10)

which is also known as Hölder’s inequality. $\square$

1.2 Proof of Theorem 3

Proof

(Theorem 3) Let $ind (\alpha )$ be an indicator function with value 1 if $\alpha$ is true and 0 otherwise. It holds

$$\begin{aligned} P(\mathbf {a}) = \mathop \int \limits _{ min _1}^{ max _1} \ldots \mathop \int \limits _{ min _m}^{ max _m} ind (x_1 \le a_1) \cdots ind (x_m\le a_m) p(x_1, \ldots , x_m) dx_1 \cdots dx_m\quad \end{aligned}$$

(11)

Using empirical data, we hence have

$$\begin{aligned} P(\mathbf {a}) = \frac{1}{k} \sum _{j=1}^k \prod _{i=1}^{m} ind (R_j^i \le a_i) , \quad \mathrm{and } \quad Q(\mathbf {a}) = \frac{1}{l} \sum _{j=1}^l \prod _{i=1}^{m} ind (S_j^i \le a_i) , \end{aligned}$$

and therefore $[ ID (p(\mathbf {A})\; ||\; q(\mathbf {A}))]^2$ equals to

$$\begin{aligned} \mathop \int \limits _{ min _1}^{ max _1} \ldots \mathop \int \limits _{ min _m}^{ max _m} \left( \frac{1}{k} \sum _{j=1}^k \prod _{i=1}^{m} ind (R_j^i \le a_i) - \frac{1}{l} \sum _{j=1}^l \prod _{i=1}^{m} ind (S_j^i \le a_i)\right) ^2 da_1 \cdots da_m\end{aligned}$$

(12)

Expanding the above term and bringing the integrals inside the sums, we have

$$\begin{aligned}&\frac{1}{k^2} \sum _{j_1=1}^k \sum _{j_2=1}^k \prod _{i=1}^{m} \mathop \int \limits _{ min _i}^{ max _i} ind \left( \max \left( R_{j_1}^i, R_{j_2}^i\right) \le a_i\right) da_i \nonumber \\&\qquad - \frac{2}{kl} \sum _{j_1=1}^k \sum _{j_2=1}^l \prod _{i=1}^{m} \mathop \int \limits _{ min _i}^{ max _i} ind \left( \max \left( R_{j_1}^i, S_{j_2}^i\right) \le a_i\right) da_i\\&\qquad + \frac{1}{l^2} \sum _{j_1=1}^{l} \sum _{j_2=1}^{l} \prod _{i=1}^{m} \mathop \int \limits _{ min _i}^{ max _i} ind \left( \max \left( S_{j_1}^i, S_{j_2}^i\right) \le a_i\right) da_i , \nonumber \end{aligned}$$

(13)

by which we arrive at the final result. $\square$

1.3 Proof of Theorem 5

Proof

(Theorem 5) Consider a discretization $dsc_i$ on dimension $X_i$ with $k_i$ macro bins $\{B_i^1, \ldots , B_i^{k_i}\}$. We have

$$\begin{aligned} L(\mathbf {M}_i, dsc_i)&\ge L_{ bid }(dsc_i(\mathbf {M}_i)) + L(\mathbf {M}_i \ominus dsc_i(\mathbf {M}_i)) \end{aligned}$$

(14)

$$\begin{aligned}&\ge \left( \sum _{j=1}^{k_i} L_{\mathbb {N}}\left( |B_i^j|\right) + \left( |B_i^j| + 1\right) \log \frac{T_i}{|B_i^j|}\right) + \sum _{j=1}^{k_i} |B_i^j| \log |B_i^j|\end{aligned}$$

(15)

$$\begin{aligned}&\ge (T_i + k_i) \log T_i - \sum _{j=1}^{k_i} \log |B_i^j|\end{aligned}$$

(16)

$$\begin{aligned}&\ge T_i \log T_i. \end{aligned}$$

(17)

Let $dsc_i^{T_i}$ be the discretization that puts each micro bin into a separate macro bin. We have

$$\begin{aligned} L\left( \mathbf {M}_i, dsc_i^{T_i}\right) = L_{\mathbb {N}}(T_i) + T_i \log c_0 + 2T_i \log T_i. \end{aligned}$$

(18)

Let $dsc_i^{ opt }$ and $dsc_i^{ gr }$ be the discretization yielded by IPD $_{ opt }$ and IPD $_{ gr }$, respectively.

Let $dsc_i$ be a discretization that merges two micro bins with a low interaction distance into the same macro bin and places each of the other micro bins into a separate macro bin. It holds that

$$\begin{aligned} L(\mathbf {M}_i, dsc_i) = L_{\mathbb {N}}(T_i - 1) + \log (T_i - 1) + (T_i - 1) \log c_0 + 2 T_i \log T_i - \log T_i. \end{aligned}$$

(19)

Thus, $L(\mathbf {M}_i, dsc_i) < L(\mathbf {M}_i, dsc_i^{T_i})$, i.e., merging two consecutive micro bins with a low interaction distance in the first place will yield an encoding cost lower than that of $dsc_i^{T_i}$. Thus, IPD $_{ gr }$ will proceed after this step. Hence, $L(\mathbf {M}_i, dsc_i^{ gr }) \le L(\mathbf {M}_i, dsc_i)$. We have $\displaystyle \frac{L(\mathbf {M}_i, dsc_i^{ gr })}{L(\mathbf {M}_i, dsc_i^{ opt })} \le \frac{L(\mathbf {M}_i, dsc_i)}{T_i \log T_i}$. This leads to

$$\begin{aligned} \frac{L\left( \mathbf {M}_i, dsc_i^{ gr }\right) }{L\left( \mathbf {M}_i, dsc_i^{ opt }\right) } \le \frac{L_{\mathbb {N}}(T_i {-} 1) + \log (T_i {-} 1) + (T_i {-} 1) \log c_0 + 2 T_i \log T_i {-} \log T_i}{T_i \log T_i}. \end{aligned}$$

(20)

Let $RHS$ be the right hand side of (20). It holds that $\lim \limits _{T_i \rightarrow \infty } RHS = 2$ as $\lim \limits _{T_i \rightarrow \infty } \frac{L_{\mathbb {N}}(T_i - 1)}{T_i \log T_i} = 0$ (Grünwald 2007). In other words, as $\displaystyle T_i \rightarrow \infty$, $\frac{L(\mathbf {M}_i, dsc_i^{ gr })}{L(\mathbf {M}_i, dsc_i^{ opt })} \le 2$. Therefore, asymptotically IPD $_{ gr }$ is a $2$-approximation algorithm of IPD $_{ opt }$. $\square$

1.4 Proof of Theorem 6

Proof

(Theorem 6) We assume that there are $(T_i - 1) \epsilon$ pairs of consecutive micro bins of $X_i$ that have low interaction distance ($0 \le \epsilon \le 1$), i.e., $(T_i - 1) (1 - \epsilon )$ pairs have a large interaction distance. We have

$$\begin{aligned} L(\mathbf {M}_i, dsc_i^1)&= \log c_0 + L_{\mathbb {N}}(T_i) + L_{\mathbb {N}}\left( (T_i - 1)(1 - \epsilon )\right) \nonumber \\&+\, (T_i - 1)(1 - \epsilon ) \log (T_i - 1) + T_i \log T_i. \end{aligned}$$

(21)

This means $\frac{L(\mathbf {M}_i, dsc_i^{ gr })}{L(\mathbf {M}_i, dsc_i^{ opt })} \le$

$$\begin{aligned} \frac{\log c_0 + L_{\mathbb {N}}(T_i) + L_{\mathbb {N}}\left( (T_i - 1)(1 - \epsilon )\right) + (T_i - 1)(1 - \epsilon ) \log (T_i - 1) + T_i \log T_i}{T_i \log T_i}. \end{aligned}$$

(22)

Let $RHS$ be the right hand side of (22). Note that $\lim \limits _{T_i \rightarrow \infty } RHS = 2 - \epsilon$. In other words, as $T_i \rightarrow \infty$, $\displaystyle \frac{L(\mathbf {M}_i, dsc_i^{ gr })}{L(\mathbf {M}_i, dsc_i^{ opt })} \le 2 - \epsilon$. $\square$