Short text topic modeling by exploring original documents

Abstract

Topic modeling for short texts faces a tough challenge, owing to the sparsity problem. An effective solution is to aggregate short texts into long pseudo-documents before training a standard topic model. The main concern of this solution is the way of aggregating short texts. A recent developed self-aggregation-based topic model (SATM) can adaptively aggregate short texts without using heuristic information. However, the model definition of SATM is a bit rigid, and more importantly, it tends to overfitting and time-consuming for large-scale corpora. To improve SATM, we propose a generalized topic model for short texts, namely latent topic model (LTM). In LTM, we assume that the observable short texts are snippets of normal long texts (namely original documents) generated by a given standard topic model, but their original document memberships are unknown. With Gibbs sampling, LTM drives an adaptive aggregation process of short texts, and simultaneously estimates other latent variables of interest. Additionally, we propose a mini-batch scheme for fast inference. Experimental results indicate that LTM is competitive with the state-of-the-art baseline models on short text topic modeling.

Appendices

Appendix

A. Derivation of Gibbs sampling LTM

We derive the Gibbs sampling equations for LTM. In terms of LTM, four latent variables of interest exist, including the topic-word distribution $\phi$, the original document-topic distribution $\theta$, the original document assignment for short texts ${\hat{z}}$ and the topic assignment for word tokens z. Thanks to the conjugate Dirichlet-Multinomial design for $\phi$ and $\theta$, we can effectively marginalize out these two variables, and turn to the joint distribution of the observation W and the two assignment variables ${\hat{z}}$ and z as follows:

$$\begin{aligned}&p\left( {W,{\hat{z}},z|\beta ,\alpha } \right) = \int {\int {p\left( {W,\phi ,\theta ,{\hat{z}},z|\beta ,\alpha } \right) \hbox {d}\phi } \hbox {d}\theta } \nonumber \\&\quad = \int {\int {\prod \limits _{k = 1}^K {Dir(\phi |\beta )} \prod \limits _{d = 1}^D {Dir(\theta |\alpha )} \prod \limits _{v = 1}^V {\prod \limits _{k = 1}^K {\prod \limits _{d = 1}^D {\phi _{kv}^{{N_{kv}}}\theta _{dk}^{{N_{dk}}}} } } \hbox {d}\phi } \hbox {d}\theta } \nonumber \\&\quad = \left( {\prod \limits _{k = 1}^K {\frac{{\prod \nolimits _{v = 1}^V {\varGamma \left( {{N_{kv}} + \beta } \right) } }}{{\varGamma \left( {{N_k} + V\beta } \right) }}\frac{{\varGamma \left( {V\beta } \right) }}{{\prod \nolimits _{v = 1}^V {\varGamma \left( \beta \right) } }}} } \right) \left( {\prod \limits _{d = 1}^D {\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{dk}} + \alpha } \right) } }}{{\varGamma \left( {{N_d} + K\alpha } \right) }}\frac{{\varGamma \left( {K\alpha } \right) }}{{\prod \nolimits _{k = 1}^K {\varGamma \left( \alpha \right) } }}} } \right) \nonumber \\&\qquad \propto \left( {\prod \limits _{k = 1}^K {\frac{{\prod \nolimits _{v = 1}^V {\varGamma \left( {{N_{kv}} + \beta } \right) } }}{{\varGamma \left( {{N_k} + V\beta } \right) }}} } \right) \left( {\prod \limits _{d = 1}^D {\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{dk}} + \alpha } \right) } }}{{\varGamma \left( {{N_d} + K\alpha } \right) }}} } \right) \nonumber \\&\quad \buildrel \varDelta \over = B\left( {{N_{kv}},{N_{k,}}\beta } \right) B\left( {{N_{dk}},{N_{d,}}\alpha } \right) \end{aligned}$$

(11)

where the fourth line in Eq. (11) follows that $\beta$ and $\alpha$ are constant; notations $B\left( {{N_{kv}},{N_{k,}}\beta } \right)$ and $B\left( {{N_{dk}},{N_{d,}}\alpha } \right)$ are used to denote $\prod \nolimits _{k = 1}^K {\frac{{\prod \nolimits _{v = 1}^V {\varGamma \left( {{N_{kv}} + \beta } \right) } }}{{\varGamma \left( {{N_k} + V\beta } \right) }}}$ and ${\prod \nolimits _{d = 1}^D {\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{dk}} + \alpha } \right) } }}{{\varGamma \left( {{N_d} + K\alpha } \right) }}} }$ for convenience.

We employ the blocked Gibbs sampling framework, where in each iteration ${\hat{z}}$ and z are alternately sampled given the other one. We derive the Gibbs sampling equations for ${\hat{z}}$ and z one by one.

Sampling equation of ${\hat{z}}$ given the current z To draw a sample of ${\hat{z}}$, following Gibbs sampling idea we consecutively draw a single original document assignment ${\hat{z}}_s$ from a posterior conditioned on all other original document assignments ${\hat{z}}^{-s}$:

$$\begin{aligned}&p\left( {{{{\hat{z}}}_s}|{{{\hat{z}}}^{ - s}},z,W, \beta ,\alpha } \right) = \frac{{p\left( {W,{\hat{z}}|z,\beta ,\alpha } \right) }}{{p\left( {W,{{{\hat{z}}}^{ - s}}|z,\beta ,\alpha } \right) }} \propto \frac{{p\left( {W,{\hat{z}}|z,\beta ,\alpha } \right) }}{{p\left( {{W^{ - s}},{{{\hat{z}}}^{ - s}}|z,\beta ,\alpha } \right) }} \nonumber \\&\quad = \frac{{p\left( {W,{\hat{z}},z|\beta ,\alpha } \right) }}{{p\left( {{W^{ - s}},{{{\hat{z}}}^{ - s}},z|\beta ,\alpha } \right) }} \propto \frac{{p\left( {W,{\hat{z}},z|\beta ,\alpha } \right) }}{{p\left( {{W^{ - s}},{{{\hat{z}}}^{ - s}},{z^{ - s}}|\beta ,\alpha } \right) }} \end{aligned}$$

(12)

By combing Eqs. (11) and (12), we have:

$$\begin{aligned}&p\left( {{{{\hat{z}}}_s} = d|{{{\hat{z}}}^{ - s}},z,W,\beta ,\alpha } \right) \propto \frac{{B\left( {{N_{kv}},{N_k},\beta } \right) B\left( {{N_{dk}},{N_d},\alpha } \right) }}{{B\left( {N_{kv}^{ - s},N_k^{ - s},\beta } \right) B\left( {N_{dk}^{ - s},N_d^{ - s},\alpha } \right) }} \nonumber \\&\quad \propto \frac{{B\left( {{N_{dk}},{N_d},\alpha } \right) }}{{B\left( {N_{dk}^{ - s},N_d^{ - s},\alpha } \right) }} \end{aligned}$$

(13)

The second line in Eq. (13) follows that the terms with respect to topic-word counts are independent of the current assignment ${\hat{z}}_s$. We then expand Eq. (13) and obtain the final Gibbs sampling equation for ${\hat{z}}$ (i.e., Eq. 1):

$$\begin{aligned}&p\left( {{{{\hat{z}}}_s} = d|{{{\hat{z}}}^{ - s}},z,W,\alpha } \right) \propto \frac{{\prod \nolimits _{j = 1}^D {\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{jk}} + \alpha } \right) } }}{{\varGamma \left( {{N_j} + K\alpha } \right) }}} }}{{\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {N_{dk}^{ - s} + \alpha } \right) } }}{{\varGamma \left( {N_d^{ - s} + K\alpha } \right) }}\prod \nolimits _{j \ne d} {\frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{jk}} + \alpha } \right) } }}{{\varGamma \left( {{N_j} + K\alpha } \right) }}} }} \nonumber \\&\quad = \frac{{\prod \nolimits _{k = 1}^K {\varGamma \left( {{N_{dk}} + \alpha } \right) } }}{{\prod \nolimits _{k = 1}^K {\varGamma \left( {N_{dk}^{ - s} + \alpha } \right) } }}\frac{{\varGamma \left( {N_d^{ - s} + K\alpha } \right) }}{{\varGamma \left( {{N_d} + K\alpha } \right) }} \nonumber \\&\quad = \frac{{\prod \nolimits _{k = 1}^K {\prod \nolimits _{n = 1}^{{N_{sk}}} {\left( {N_{dk}^{ - s} + n - 1 + \alpha } \right) } } }}{{\prod \nolimits _{n = 1}^{{N_s}} {\left( {N_d^{ - s} + n - 1 + K\alpha } \right) } }} \end{aligned}$$

(14)

The third line in Eq. (14) follows the fact ($m>n$):

$$\begin{aligned} \frac{{\varGamma \left( n \right) }}{{\varGamma \left( m \right) }} = \frac{{\varGamma \left( n \right) }}{{\varGamma \left( {n + 1} \right) }}\frac{{\varGamma \left( {n + 1} \right) }}{{\varGamma \left( {n + 2} \right) }}\frac{{\varGamma \left( {n + 2} \right) }}{{\varGamma \left( {n + 3} \right) }} \cdots \frac{{\varGamma \left( {m - 1} \right) }}{{\varGamma \left( m \right) }} = \frac{1}{{\prod \nolimits _{i = 1}^{m - n} {\left( {n + i - 1} \right) } }} \end{aligned}$$

Sampling equation of z given the current ${\hat{z}}$ We now turn to the sampling of z. Note that given ${\hat{z}}$, the case is completely equivalent to the standard LDA Gibbs sampling. Similar with the derivations of Eqs. (12) and (13), the posterior of a single topic assignment $z_{dn}$ conditioned on all other topic assignments $z^{-dn}$ is given by:

$$\begin{aligned}&p\left( {{z_{dn}}|{\hat{z}},z,W,\beta ,\alpha } \right) \propto \frac{{p\left( {W,{\hat{z}},z|\beta ,\alpha } \right) }}{{p\left( {{W^{ - s}},{{{\hat{z}}}^{ - s}},{z^{ - s}}|\beta ,\alpha } \right) }} \nonumber \\&\quad = \frac{{B\left( {{N_{kv}},{N_k},\beta } \right) B\left( {{N_{dk}},{N_d},\alpha } \right) }}{{B\left( {N_{kv}^{ - s},N_k^{ - s},\beta } \right) B\left( {N_{dk}^{ - s},N_d^{ - s},\alpha } \right) }} \end{aligned}$$

(15)

By expanding Eq. (15), we derive the final Gibbs sampling equation for z (i.e., Eq. 2):

$$\begin{aligned}&p\left( {{z_{dn}} = k|{\hat{z}},{z^{ - dn}},W,\beta ,\alpha } \right) \propto \frac{{\frac{{\varGamma \left( {{N_{k{w_{dn}}}} + \beta } \right) }}{{\varGamma \left( {{N_k} + V\beta } \right) }}\frac{{\varGamma \left( {{N_{k{w_{dn}}}} + \beta } \right) }}{{\varGamma \left( {{N_k} + V\beta } \right) }}}}{{\frac{{\varGamma \left( {N_{k{w_{dn}}}^{ - dn} + \beta } \right) }}{{\varGamma \left( {N_k^{ - dn} + V\beta } \right) }}\frac{{\varGamma \left( {N_{dk}^{ - dn} + \alpha } \right) }}{{\varGamma \left( {N_d^{ - dn} + K\alpha } \right) }}}} \nonumber \\&\quad = \frac{{N_{k{w_{dn}}}^{ - dn} + \beta }}{{N_k^{ - dn} + V\beta }}\frac{{N_{dk}^{ - dn} + \alpha }}{{N_d^{ - dn} + K\alpha }} \nonumber \\&\qquad \propto \frac{{N_{k{w_{dn}}}^{ - dn} + \beta }}{{N_k^{ - dn} + V\beta }}\left( {N_{dk}^{ - dn} + \alpha } \right) \end{aligned}$$

(16)

The third line in Eq. (16) follows that the term $(N_d^{ - dn} + K\alpha )$ is independent of the current topic assignment $z_{dn}$.