PMIVec: a word embedding model guided by point-wise mutual information criterion

Abstract

Word embedding aims to represent each word with a dense vector which reveals the semantic similarity between words. Existing methods such as word2vec derive such representations by factorizing the word–context matrix into two parts, i.e., word vectors and context vectors. However, only one part is used to represent the word, which may damage the semantic similarity between words. To address this problem, this paper proposes a novel word embedding method based on point-wise mutual information criterion (PMIVec). Our method explicitly learns the context vector as the final word representation for each word, while discarding the word vector. To avoid the damage of semantic similarity between words, we normalize the word vector during the training process. Moreover, this paper uses point-wise mutual information to measure the semantic similarity between words, which is more consistent with human intuition on semantic similarity. Experiments on public data sets show that our PMIVec model can consistently outperform state-of-the-art models.

Appendix

1.1 Proof of Theorem 1

There are two steps to prove Theorem 1. The first step is to formulate the loss function of SG model by applying the noise contrastive estimation (NCE) method [7] to equation (3) and (4). The second step is to reveal that the simplification made by SG’s negative sampling technique will lead to property (5).

1.1.1 Step 1

To find the assumed optimal solutions $\{{\mathbf {O}}_i,{\mathbf {I}}_k\}_{i,k=1,\ldots ,V}$, and yet to avoid directly computing the denominator of Eq. (4), which will be denoted as $Z_{k}$, NCE proposes to treat it as a constant number to be estimated, and one can estimate $\{{\mathbf {O}}_i,{\mathbf {I}}_k,Z_k\}_{i,k=1,\ldots ,V}$ via solving a supervised learning task.

One can draw $T_d$ real samples from $p(\cdot |w_{k})$, and $K*T_d$ noise samples from the known noise distribution $p_{n}(\cdot )$, where $p_{n}(\cdot )$ can be chosen to be any distribution and K is a constant like 5. Each sample will have a label y, and $y=1$ stands for real sample; $y=0$ stands for noise sample. Then, one can mix these samples together and pick one of them, asking whether it is a real sample or not.

One can train a logistic regression model, whose parameters are $\{{\mathbf {O}}_i,{\mathbf {I}}_k,Z_k\}_{i,k=1,\ldots ,V}$, to discriminate the real samples from the noise samples. For a real sample $w_{i}$, the logistic regression model’s prediction on it being true is

$$\begin{aligned} \begin{array}{ll} &{}p(y=1|w_i;w_k)\\ &{}=\sigma \left[ \log (\exp \left( \left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>\right) )-\log (Z_{k})-\log (Kp_{n}(w_{i}))\right] . \end{array} \end{aligned}$$

According to NCE’s conclusion, if one treats $Z_{k}$ as an additional scalar parameter that can be optimized, then the following optimization problems (minimizing the cross entropy) will have the same solution as to (3)

$$\begin{aligned} -{\mathbf {E}}_{i\sim p_{d}}\log p(1|w_i;w_k)-K{\mathbf {E}}_{t\sim p_{n}}\log p(0|w_t;w_k) \end{aligned}$$

(15)

where $\forall k=1,\ldots ,V,$

$$\begin{aligned} p(1|w_i;w_k)=\sigma \left[ \left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>-\log Z_{k}-\log Kp_{n}(w_{i})\right] , \end{aligned}$$

and $p_{d}$ represents the true distribution of $p(\cdot |w_{k})$. One can sample $w_i$ from $p_d$ with word window’s help.

1.1.2 Step 2

The following part will show how the negative-sampling technique adopted by the skip-gram model can lead to Eq. (5).

The SG model simplifies the calculation of $p(1|w_i;w_k)$ by omitting $\log Z_{k}$ and $\log Kp_{n}(w_{i})$. This means that the logistic model assumes $Kp_{n}(w_{i})=1$ [5], i.e.,

$$\begin{aligned} \begin{array}{rl} p\left( y=1|w_i;w_k\right) &{}=\sigma (\left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>)=\frac{\exp (\left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>)}{\exp (\left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>)+1},\\ {\hat{p}}_{d}\left( y=1|w_i;w_k\right) &{}= \frac{p_{d}(w_{i}|w_{k})}{p_{d}(w_{i}|w_{k}) + Kp_{n}(w_{i})}. \\ \end{array} \end{aligned}$$

where ${\hat{p}}_{d}(1|w_i;w_k)$ represents the true probability of sample $w_{i}$ being true, and it can be derived by applying the Bayesian formula to it. It is easy to show that (15) is equivalent to

$$\begin{aligned} \sum \limits _{i}\rho *KL\left\{ {\hat{p}}_{d}\left( y|w_i;w_k\right) \Vert p\left( y|w_i;w_k\right) \right\} , \end{aligned}$$

where $\rho =p_{d}(w_i;w_k)+Kp_{n}(w_{j})$. Obviously, the above objective will equal to 0 iff

$$\begin{aligned} {\hat{p}}_{d}\left( y|w_i;w_k\right) =p\left( y|w_i;w_k\right) ,\ y=1,0. \end{aligned}$$

(16)

Noticing that SG chooses $p(w_{k})$ as the noise distribution, and with some simple rearrangement, the Eq. (16) implies

$$\begin{aligned} \left<{\mathbf {O}}_{i},{\mathbf {I}}_{k}\right>=\mathrm{PMI}(w_{i},w_{k})-\log K. \end{aligned}$$

1.2 Proof of Theorem 2

There are two steps to prove Theorem 2. The first step is to show the following equations hold for the assumed solutions $\{{\mathbf {O}}_i,{\mathbf {I}}_k\}_{i,k=1,\ldots ,V}$

$$\begin{aligned} \log \left( p(w_i)\right) + \log \left( {\hat{Z}}\right)&={\bar{\alpha }}_i*\left\| {\mathbf {O}}_i\right\| _2^2, \end{aligned}$$

(17)

$$\begin{aligned} \log \left( p(w_j)\right) + \log \left( {\hat{Z}}\right)&={\bar{\alpha }}_j*\left\| {\mathbf {O}}_j\right\| _2^2, \end{aligned}$$

(18)

$$\begin{aligned} \log \left( p(w_i,w_j)\right) + \log \left( {\hat{Z}}^\prime \right)&={\bar{\alpha }}_{ij}*\Vert {\mathbf {O}}_i+{\mathbf {O}}_j\Vert _2^2, \end{aligned}$$

(19)

where ${\hat{Z}}$’s definition can be found in the Eq. (10), ${\hat{Z}}^\prime$ is the mean value of all ${\hat{Z}}_k^\prime$, and ${\hat{Z}}_k^\prime$ is defined in the Eq. (12). ${\bar{\alpha }}_i,{\bar{\alpha }}_j$, and ${\bar{\alpha }}_{ij}$ are both constant number. The second step is to show that

$$\begin{aligned} \begin{array}{rl} {\hat{Z}}^\prime &{}\approx {\hat{Z}}^2,\\ {\bar{\alpha }}_i&{}\approx {\bar{\alpha }}_j\approx {\bar{\alpha }}_{ij}, \end{array} \end{aligned}$$

and therefore, (19), (18), and (17) will prove this theorem.

1.2.1 Step 1

Similar to the analysis in the PMIVec section’s sketch proof, the assumed solution $\{{\mathbf {O}}_i,{\mathbf {I}}_k\}_{i,k=1,\ldots ,V}$ satisfies

$$\begin{aligned} \begin{array}{rl} \sum \limits _{k=1}^{V}p\left( w_i,w_k\right) Z_k &{}=\sum \limits _{k=1}^{V} \exp \left( \left\| {\mathbf {O}}_i\right\| _2^2\cos \left( \theta _{ik}\right) \right) p(w_k),\\ \sum \limits _{k=1}^{V}p\left( w_i,w_j,w_k\right) Z_k^\prime &{}=\\ \sum \limits _{k=1}^{V}&{} \exp \left( \left\| {\mathbf {O}}_i+{\mathbf {O}}_j\right\| _2^2\cos (\theta _{(ij)k})\right) p(w_k).\\ \end{array} \end{aligned}$$

Then the $\alpha _{i}$ is the mean value of $\cos (\theta _{ik})$, $\alpha _{j}$ is the mean value of $\cos (\theta _{jk})$, and $\alpha _{ij}$ is the mean value of $\cos (\theta _{(ij)k})$.

The log value of both sides of these equations leads to Eqs. (17), (18), and (19).

1.2.2 Step 2

It is obvious that ${\bar{\alpha }}_i\approx {\bar{\alpha }}_j\approx {\bar{\alpha }}_{ij}$ because of the assumption about $\theta _{ik},\theta _{jk}$, and $\theta _{(ij)k}$.

Noticing that each term of $Z^{\prime }_k$ is bigger than each term of $Z_k^2$, and meanwhile that the total number of $Z^{\prime }_k$ is smaller than the total number of $Z_k^2$. Therefore, $Z^{\prime }_k\approx Z^{2}_k$. This leads to the conclusion that ${\hat{Z}}^\prime \approx {\hat{Z}}^2$.