Data-dependent dissimilarity measure: an effective alternative to geometric distance measures

Abstract

Nearest neighbor search is a core process in many data mining algorithms. Finding reliable closest matches of a test instance is still a challenging task as the effectiveness of many general-purpose distance measures such as $\ell _p$-norm decreases as the number of dimensions increases. Their performances vary significantly in different data distributions. This is mainly because they compute the distance between two instances solely based on their geometric positions in the feature space, and data distribution has no influence on the distance measure. This paper presents a simple data-dependent general-purpose dissimilarity measure called ‘$m_p$-dissimilarity’. Rather than relying on geometric distance, it measures the dissimilarity between two instances as a probability mass in a region that encloses the two instances in every dimension. It deems two instances in a sparse region to be more similar than two instances of equal inter-point geometric distance in a dense region. Our empirical results in k-NN classification and content-based multimedia information retrieval tasks show that the proposed $m_p$-dissimilarity measure produces better task-specific performance than existing widely used general-purpose distance measures such as $\ell _p$-norm and cosine distance across a wide range of moderate- to high-dimensional data sets with continuous only, discrete only, and mixed attributes.

Appendices

Appendix 1: Probabilistic interpretation of $m_p$

The formulation of $m_p(\mathbf{x}, \mathbf{y})$ (Eq. 8) has a probabilistic interpretation. The simplest form of data-dependent dissimilarity measure is to define an M-dimensional region $R(\mathbf{x}, \mathbf{y})$ that encloses $\mathbf{x}$ and $\mathbf{y}$, and to estimate the probability of a randomly selected point $\mathbf{t}$ from the distribution of data, $\phi (\mathbf{x})$, falling in $R(\mathbf{x}, \mathbf{y})$, $P(\mathbf{t} \in R(\mathbf{x},\mathbf{y})|\phi (\mathbf{x}))$. Let $R(\mathbf{x}, \mathbf{y})$ has length of $R_i(\mathbf{x}, \mathbf{y})$ in dimension i. Assuming that the dimensions are independent, $P(\mathbf{t} \in R(\mathbf{x},\mathbf{y})|\phi (\mathbf{x}))$ can be approximated as:

$$\begin{aligned} P(\mathbf{t} \in R(\mathbf{x},\mathbf{y})|\phi (\mathbf{x})) \approx \prod _{i=1}^M P(t_i \in R_i(\mathbf{x},\mathbf{y})|\phi _i(\mathbf{x})) \end{aligned}$$

(14)

where $P(t_i\in R_i(\mathbf{x},\mathbf{y})|\phi _i(\mathbf{x}))$ is the probability of $t_i$ falling in $R_i(\mathbf{x},\mathbf{y})$ for dimension i.

The approximation in Eq. 14 is sensitive to outliers. An approximation which is tolerant to outliers can be estimated by replacing the product with the summation [23]. The sum-based approximation relates to the probability of $\mathbf{t}$ in Eq. 14 under the following outlier model. Consider a data generation process in which in order to sample $t_i$, a coin with probability of turning head $(1-\epsilon )$ is flipped. If the coin turns head, $t_i$ is drawn from the distribution of data in dimension i, $\phi _i(\mathbf{x})$, where the probability of sampling $t_i$ is $P_i(t_i|\phi _i(\mathbf{x}))$, otherwise it is sampled from the uniform distribution with probability 1 / A, and A is a constant.

Lemma 1

[23] Under the data generation process described above, the probability of a data point $P'(\cdot )$ can be approximated as

$$\begin{aligned} P'(\mathbf{t}|\phi (\mathbf{x}),\epsilon ) \approx C_1 + C_2 \times \sum _{i=1}^M P_i(t_i|\phi _i(\mathbf{x})) \end{aligned}$$

where $C_1$ and $C_2$ are constants.

Proof

Under the outlier model, the probability of generating the value of the i’th dimension $t_i$ is

$$\begin{aligned} P'(t_i|\phi (\mathbf{x}),\epsilon ) = \epsilon /A + (1-\epsilon )P(t_i|\phi _i(\mathbf{x})) \end{aligned}$$

(15)

We assume that each dimension is generated independently, hence

$$\begin{aligned} P'(\mathbf{t}|\phi (\mathbf{x}),\epsilon )\approx & {} \displaystyle \prod _{i=1}^M P'(t_i|\phi (\mathbf{x}),\epsilon ) = \displaystyle \prod _{i=1}^M \left( \epsilon /A + (1-\epsilon ) P(t_i|\phi _i(\mathbf{x}))\right) \\= & {} (\epsilon /A)^M + (\epsilon /A)^{M-1}(1-\epsilon )\sum _{i=1}^M P(t_i|\phi _i(\mathbf{x})) + O\left( (1-\epsilon )^2\right) \end{aligned}$$

In the extreme case where the probability of generating $t_i$ from the uniform distribution (i.e., the outlier component) is high, i.e., $\epsilon$ is close to 1, only the first two terms matter. Assuming $C_1 := (\epsilon /A)^M$ and $C_2 := (\epsilon /A)^{M-1} (1-\epsilon )$, the lemma follows. $\square$

In addition to the above approximation given by Minka [23], we propose that the chance of $t_i$ being drawn from the outlier model can be further reduced by sampling from $\phi _i(\mathbf{x})^p$, $p>1$ when coin turns up head in the above mentioned data generation process. The probability of sampling $t_i$ from $\phi _i(\mathbf{x})^p$ is $\frac{P(t_i|\phi _i(\mathbf{x}))^p}{Z_{i,p}}$, where $P(\cdot )^p$ is the probability of a random event occurring in p successive trials and $Z_{i,p}$ is the normalization constant to ensure the total probability sums up to 1 in the $i^{th}$ dimension.

Lemma 2

Under the data generation process of sampling from exponential distribution described above, the probability of a data point $P''(\cdot )$ can be approximated as

$$\begin{aligned} P''(\mathbf{t}|\phi (\mathbf{x}),\epsilon ,p) \approx C_1 + C_2 \times \sum _{i=1}^M \frac{P_i(t_i|\phi _i(\mathbf{x}))^p}{Z_{i,p}} \end{aligned}$$

where $C_1$, $C_2$, and $\{Z_{i,p}\}_{i=1}^M$ are constants.

Proof

This follows from Lemma 1 by drawing $t_i$ from $\phi _i(\mathbf{x})^p$ $p>1$ when coin turns up head in the data generation process. $\square$

As a result of Lemma 2 (by considering the outlier tolerant model), $P(\mathbf{t} \in R(\mathbf{x},\mathbf{y}))$ can be approximated as:

$$\begin{aligned} P(\mathbf{t} \in R(\mathbf{x},\mathbf{y})) \approx \displaystyle C_1 + C_2 \times \sum _{i=1}^M \frac{P_i(t_i \in R_i(\mathbf{x},\mathbf{y}))^p}{Z_{i,p}} \end{aligned}$$

(16)

Note that $P(\mathbf{t} \in R(\mathbf{x},\mathbf{y}))$ is a data-dependent dissimilarity measure for $\mathbf{x}$ and $\mathbf{y}$. All the constants on RHS of Eq. 16 are independent of $\mathbf{x}$ and $\mathbf{y}$ and they are just the scaling factors of the dissimilarity measure. Particularly, in order to find the nearest neighbor of $\mathbf{x}$ among a collection of data instances, the only important term in the measure is $\sum _{i=1}^M P_i(t_i \in R_i(\mathbf{x},\mathbf{y}))^p$. The constants can be ignored as they do not change the ranking of data points. Hence, by ignoring the constants in Eq. 16, $m_p(\mathbf{x}, \mathbf{y})$ can be expressed as its rescaled version as follows:

$$\begin{aligned} m_p(\mathbf{x},\mathbf{y}) = \displaystyle \left( \frac{1}{M}\sum _{i=1}^M P_i\left( t_i \in R_i(\mathbf{x}, \mathbf{y})\right) ^p\right) ^\frac{1}{p} \end{aligned}$$

(17)

where the outer power of $\frac{1}{p}$ is just a rescaling factor and $\frac{1}{M}$ is a constant.

In practice, $P_i\left( t_i\in R_i(\mathbf{x}, \mathbf{y})\right)$ can be estimated from D as:

$$\begin{aligned} {\hat{P}}_i\left( t_i\in R_i(\mathbf{x}, \mathbf{y})\right) = \frac{|R_i(\mathbf{x}, \mathbf{y})|}{N} \end{aligned}$$

(18)

Hence, Eqs. 17 and 18 lead to $m_p$ defined in Eq. 8.

Appendix 2: Analysis of concentration and hubness

In order to examine the concentration and hubness of the three dissimilarity measures $m_2$, $\ell _2$ and $d_{cos}$ in different data distributions with the increase in the number of dimensions, we used synthetic data sets with uniform (each dimension is uniformly distributed between [0,1]) and normal (each dimension is normally distributed with zero mean and unit variance) distributions with $M=10$ and $M=200$. Feature vectors were normalized to be in unit range in each dimension.

1.1 Concentration

The relative contrast between the nearest and farthest neighbor is computed for all $N=1000$ instances in each data set using $m_2$, $\ell _2$ and $d_{cos}$. The relative contrast for each instance in uniform and normal distributions with $M=10$ and $M=200$ are shown in Fig. 5.

Fig. 5

Relative contrast $\left( \frac{dmax(\mathbf{x},d)-dmin(\mathbf{x},d)}{dmin(\mathbf{x},d)}\right)$ of $m_2$, $\ell _2$ and $d_{cos}$. Note that x axis is instance id and corresponding y axis value is the relative contrast of that instance

The relative contrast of all three measures decreased substantially (note that the y-axes have different scales in Fig. 5) when the number of dimensions was increased from $M=10$ to $M=200$ in both distributions. It is interesting to note that $m_2$ has the least relative contrast in both distributions with $M=10$ and $M=200$; and $d_{cos}$ has the maximum relative contrast in all cases. The relative contrasts of $\ell _2$ and $m_2$ are almost the same except in the case of normal ($M=200$), where the relative contrast of $\ell _2$ is slightly higher than that of $m_2$ for many instances.

This suggests that $m_2$ is more concentrated than $\ell _2$ and $d_{cos}$. Even in real-world data sets, we observed that $m_2$ is more concentrated than $\ell _2$ and $d_{cos}$.

Fig. 6

The $O_5$ distributions of $m_2$, $\ell _2$ and $d_{cos}$ in synthetic data sets. Note that x axis is in the log scale hence x axis value is $\log (O_5+1)$ to consider the case of $O_5=0$

1.2 Hubness

In order to examine the hubness phenomenon, 5-Occurrences of each instance $\mathbf{x}\in D$ is estimated, i.e., $O_5(\mathbf{x}) = |\{\mathbf{y}: \mathbf{x}\in N_5(\mathbf{y})\}|$, where $N_5(\mathbf{y})$ is the set of 5-NN of $\mathbf{y}$. Then, the $O_5$ distribution is plotted for each measure ($m_2$, $\ell _2$ and $d_{cos}$) in all four synthetic data sets which is shown in Fig. 6.

The $O_5$ distributions of all three measures become skewed when the number of dimensions was increased from $M=10$ to $M=200$ in both distributions. It is interesting to note that the $O_5$ distributions of $m_2$ in uniform and normal distributions are almost similar for both $M=10$ and $M=200$, whereas those of $\ell _2$ and $d_{cos}$ in the case of normal distribution are more skewed than those in uniform distribution for both $M=10$ and $M=200$. Note that the $O_5$ distributions of $m_2$ and $\ell _2$ in uniform distribution are similar for both $M=10$ and $M=200$. This is because of the fact that $m_2$ is proportional to $\ell _2$ under uniform distribution (also reflected in Fig. 2a). In the case of normal distribution and $M=200$, the $O_5$ distribution of $m_2$ is less skewed than those of $\ell _2$ and $d_{cos}$. There are 361 and 348 (out of 1000) instances with $O_5=0$ (which do not occur in the 5-NN set of any other instance) in the case of $\ell _2$ and $d_{cos}$, respectively; whereas there are only 161 instances with $O_5=0$ in the case of $m_2$. Similarly, the most popular nearest neighbors using $\ell _2$ and $d_{cos}$ have $O_5=146$ and 152, respectively; whereas the most popular nearest neighbor using $m_2$ has $O_5=69$.

Table 10 Standard error of accuracies of k-NN classification ($k=5$) over a tenfold cross-validation. Average classification accuracy is presented in Table 4 in Sect. 4.1

We observed similar behavior in many real-world data sets as well where the $O_5$ distribution of $m_2$ is less skewed than that of $\ell _2$ and $d_{cos}$.

Table 11 Standard error of P@10 over N queries. Average P@10 is presented in Table 6 in Sect. 4.2

Table 12 Average accuracy of 5-NN classification over a tenfold cross-validation

Appendix 3: Standard error

Table 10 shows the standard error of classification accuracies (in %) of k-NN classification ($k=5$) over a tenfold cross-validation (average classification accuracy is presented in Table 4 in Sect. 4.1).

Table 11 shows the standard error of precision at top 10 retrieved results (P@10) over N queries in content-based multimedia information retrieval (average P@10 is presented in Table 6 in Sect. 4.2).

Appendix 4: Comparison with geometric distance measures after dimensionality reduction

Average 5-NN classification accuracies over a tenfold cross-validation of $d_{cos}, \ell _{0.5}$ and $\ell _2$ before and after dimensionality reduction through PCA along with those of $m_{0.5}$ and $m_2$ in the original space in 16 out of 22 data sets with continuous only attributes are provided in Table 12. With PCA, the number of dimensions was reduced by projecting data in the lower-dimensional space defined by the principal components capturing 95% of the variance in data. The principal components were computed by the eigen decomposition of the correlation matrix of the training data to ensure that the projection is robust to scale differences in the original dimensions. Note that PCA did not complete in 24 h in the remaining six data sets with $M>5000$: New3s (26,832), Ohscal (11,465), Arcene (10,000), Wap (8460), R52 (7369) and NG20 (5489).