A topic model for co-occurring normal documents and short texts

Abstract

User comments, as a large group of online short texts, are becoming increasingly prevalent with the development of online communications. These short texts are characterized by their co-occurrences with usually lengthier normal documents. For example, there could be multiple user comments following one news article, or multiple reader reviews following one blog post. The co-occurring structure inherent in such text corpora is important for efficient learning of topics, but is rarely captured by conventional topic models. To capture such structure, we propose a topic model for co-occurring documents, referred to as COTM. In COTM, we assume there are two sets of topics: formal topics and informal topics, where formal topics can appear in both normal documents and short texts whereas informal topics can only appear in short texts. Each normal document has a probability distribution over a set of formal topics; each short text is composed of two topics, one from the set of formal topics, whose selection is governed by the topic probabilities of the corresponding normal document, and the other from a set of informal topics. We also develop an online algorithm for COTM to deal with large scale corpus. Extensive experiments on real-world datasets demonstrate that COTM and its online algorithm outperform state-of-art methods by discovering more prominent, coherent and comprehensive topics.

Appendix: Details of deriving the collapsed gibbs sampling algorithm

Given the full posterior distribution in (1), we can easily get the full conditional posterior distributions for Θ, Φ, Ψ, ξ and P.

For 𝜃 _d ,d ∈{1, 2,...,D}, its full conditional posterior distribution is:

$$ \begin{array}{ll} f(\boldsymbol{\theta_{d}}\mid \cdot) \propto \prod\limits_{k=1}^{K}(\theta_{dk})^{l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha-1}. \end{array} $$

(17)

For ϕ _k ,k ∈{1, 2,...,K}, its full conditional posterior distribution is:

$$ \begin{array}{ll} f(\boldsymbol{\phi_{k}}\mid \cdot) \propto \prod\limits_{v=1}^{V}(\phi_{kv})^{l_{kv}^{(2)}+g_{kv}^{(2)}+\beta-1}. \end{array} $$

(18)

For ψ _j ,j ∈{1, 2,...,J}, its full conditional posterior distribution is:

$$ \begin{array}{ll} f(\boldsymbol{\psi_{j}}\mid \cdot) \propto \prod\limits_{v=1}^{V}(\psi_{jv})^{g_{jv}^{(3)}+\beta-1}. \end{array} $$

(19)

For ξ, its full conditional posterior distribution is:

$$ \begin{array}{ll} f(\boldsymbol{\xi}\mid \cdot) \propto \prod\limits_{j=1}^{J}(\epsilon_{j})^{h_{j}+\epsilon-1}. \end{array} $$

(20)

For p _{d c} ,d ∈{1, 2,...,D},c ∈{1, 2,...,C _d }, its full conditional posterior distribution is:

$$ \begin{array}{ll} f(p_{dc}\mid \cdot)\propto (p_{dc})^{s_{dc}^{(1)}+\gamma-1}(1-p_{dc})^{s_{dc}^{(2)}+\gamma-1}. \end{array} $$

(21)

Noting the posterior distributions of Θ, Φ, Ψ, ξ and P are all Dirichlet, conjugate with their priors, we can develop a collapsed Gibbs sampling algorithm by integrating out these parameters from the posterior distribution.

To describe this procedure, we start with introducing the Dirichlet distribution. Suppose X = (X ₁,...,X _K )^T, following a Dirichlet distribution with parameter α = (α ₁,...,α _K )^T. The probability density function of X is

$$ \begin{array}{ll} f(\boldsymbol{X}|\boldsymbol{\alpha})=f(X_{1},...,X_{K}|\alpha_{1},...,\alpha_{K})=\frac{\Gamma\left( {\sum}_{i=1}^{K} \alpha_{i}\right)}{{\prod}_{i=1}^{K} {\Gamma}(\alpha_{i})}{\prod}_{i=1}^{K} X_{i}^{\alpha_{i}-1}. \end{array} $$

(22)

Since the integral of f(X|α) is equal to 1, we can get

$$ \begin{array}{ll} &\int \left\{{\prod}_{i=1}^{K} X_{i}^{\alpha_{i}-1}\right\} dX_{1}...dX_{K} = \frac{{\prod}_{i=1}^{K} {\Gamma}(\alpha_{i})}{\Gamma\left( {\sum}_{i=1}^{K} \alpha_{i}\right)}. \end{array} $$

(23)

Similarly, given the conditional posterior distribution of Θ is Dirichlet, as described in (17), we can integrate it out and get:

$$ \begin{array}{ll} &\int f(\boldsymbol{\Theta}\mid \cdot) d\boldsymbol{\Theta} = {\prod}_{d=1}^{D} \int f(\boldsymbol{\theta_{d}}\mid \cdot) d\boldsymbol{\theta_{d}}\\ \propto &{\prod}_{d=1}^{D}\int \left\{\prod\limits_{k=1}^{K}(\theta_{dk})^{l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha-1}\right\} d\theta_{d1}...d\theta_{dK} ={\prod}_{d=1}^{D}\frac{{\prod}_{i=1}^{K} {\Gamma}\left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)}{\Gamma\left\{{\sum}_{k=1}^{K} (l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha)\right\}}. \end{array} $$

(24)

Then we integrate out Φ, Ψ, ξ and P similarly and get the following results:

$$ \begin{array}{ll} \int f(\boldsymbol{\Phi}\mid \cdot) d\boldsymbol{\Phi} &= {\prod}_{k=1}^{K} \int f(\boldsymbol{\phi_{k}}\mid \cdot) d\boldsymbol{\phi_{k}} \propto {\prod}_{k=1}^{K}\frac{{\prod}_{v=1}^{V} {\Gamma}\left( l_{kv}^{(2)}+g_{kv}^{(2)}+\beta\right)}{\Gamma\left\{{\sum}_{v=1}^{V} \left( l_{kv}^{(2)}+g_{kv}^{(2)}+\beta\right)\right\}}\\[-1pt] \int f(\boldsymbol{\Psi}\mid \cdot) d\boldsymbol{\Psi} &= {\prod}_{j=1}^{J} \int f(\boldsymbol{\psi_{j}}\mid \cdot) d\boldsymbol{\psi_{j}} \propto {\prod}_{j=1}^{J}\frac{{\prod}_{v=1}^{V} {\Gamma}\left( g_{jv}^{(3)}+\beta\right)}{\Gamma\left\{{\sum}_{v=1}^{V} \left( g_{jv}^{(3)}+\beta\right)\right\}}\\[-1pt] \int f(\boldsymbol{\xi}\mid \cdot) d\boldsymbol{\xi} &\propto \frac{{\prod}_{j=1}^{J} {\Gamma}(h_{j}+\epsilon)}{\Gamma\left\{{\sum}_{j=1}^{J} (h_{j}+\epsilon)\right\}}\\[-1pt] \int f(\boldsymbol{P}\mid \cdot) d\boldsymbol{P} &= {\prod}_{d=1}^{D}{\prod}_{c=1}^{C_{d}} \int f(p_{dc}\mid \cdot) dp_{dc} \propto {\prod}_{d=1}^{D}{\prod}_{c=1}^{C_{d}}\frac{\Gamma\left( s_{dc}^{(1)}+s_{dc}^{(2)}+\gamma\right)}{\Gamma\left( s_{dc}^{(1)}+\gamma\right){\Gamma}\left( s_{dc}^{(2)}+\gamma\right)} \end{array} $$

(25)

By integrating out Θ, Φ, Ψ, ξ and P, the full posterior distribution in (1) can be simplified as:

$$ \begin{array}{ll} &f(\boldsymbol{z},\boldsymbol{b},\boldsymbol{x},\boldsymbol{y} \mid \boldsymbol{w},\alpha,\beta,\gamma,\epsilon)\\ =&\int f(\boldsymbol{z},\boldsymbol{b},\boldsymbol{P},\boldsymbol{x},\boldsymbol{y},\boldsymbol{\Theta},\boldsymbol{\Phi},\boldsymbol{\Psi},\boldsymbol{\xi} \mid \boldsymbol{w},\alpha,\beta,\gamma,\epsilon) d\boldsymbol{\Theta} d\boldsymbol{\Phi} d\boldsymbol{\Psi} d\boldsymbol{\xi} d\boldsymbol{P} \\ \propto &{\prod}_{d=1}^{D}\frac{{\prod}_{i=1}^{K} {\Gamma}\left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)}{\Gamma\left\{{\sum}_{k=1}^{K} \left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)\right\}}\times {\prod}_{k=1}^{K}\frac{{\prod}_{v=1}^{V} {\Gamma}\left( l_{kv}^{(2)}+g_{kv}^{(2)}+\beta\right)}{\Gamma\left\{{\sum}_{v=1}^{V} \left( l_{kv}^{(2)}+g_{kv}^{(2)}+\beta\right)\right\}}\\[-1pt] \times\! &{\prod}_{j=1}^{J}\frac{{\prod}_{v=1}^{V} {\Gamma}\left( g_{jv}^{(3)}+\beta\right)} {\Gamma\left\{{\sum}_{v=1}^{V} (g_{jv}^{(3)}\,+\,\beta)\right\}} \!\times\! \frac{{\prod}_{j=1}^{J} {\Gamma}(h_{j}+\epsilon)}{\Gamma\left\{{\sum}_{j=1}^{J} (h_{j}\,+\,\epsilon)\right\}} \!\times\! {\prod}_{d=1}^{D}{\prod}_{c=1}^{C_{d}}\frac{\Gamma\left( s_{dc}^{(1)}\,+\,s_{dc}^{(2)}\,+\,\gamma\right)}{\Gamma\left( s_{dc}^{(1)}\,+\,\gamma){\Gamma}(s_{dc}^{(2)}\,+\,\gamma\right)}. \end{array} $$

(26)

Thus, we can use the collapsed Gibbs sampling and only need to update z, x, y and b in each iteration. We then derive the conditional posterior distributions of z, x, y and b from (26).

Specifically, for the n th word in normal document d, z _{d n} = k only influences $l_{dk}^{(1)}$ and $l_{kw_{dn}}^{(2)}$ in (26). Let z _{−d n} denote z excluding z _{d n} , and the full conditional distribution of z _{d n} can be derived as:

$$ \begin{array}{lll} &&f(z_{dn}=k \mid \cdot)=\frac{f(z_{dn}=k,\boldsymbol{z}_{-dn} \mid \cdot)}{f(\boldsymbol{z}_{-dn} \mid \cdot)}\\ &&\propto\frac{\Gamma\left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)} {\Gamma\left( l_{dk;-dn}^{(1)}+g_{dk}^{(1)}+\alpha\right)}/ \frac{\Gamma\left\{{\sum}_{k' \neq k} \left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)+\left( l_{dk;-dn}^{(1)}+g_{dk}^{(1)}+\alpha\right)\right\}} {\Gamma\left\{{\sum}_{k' \neq k} \left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)+\left( l_{dk}^{(1)}+g_{dk}^{(1)}+\alpha\right)\right\}}\\ &&\times\frac{\Gamma\left( l_{kw_{dn}}^{(2)}\,+\,g_{kw_{dn}}^{(2)}+\beta\right)}{\Gamma\left( l_{kw_{dn};-dn}^{(2)}+g_{kw_{dn}}^{(2)}+\beta\right)} / \frac{\Gamma\left\{{\sum}_{v\neq w_{dn}}\left( l_{kv}^{(2)}+g_{kv}^{(2)}\!+\beta\right)\,+\,\left( l_{kw_{dn};-dn}^{(2)}\,+\,g_{kw_{dn}}^{(2)}\,+\,\beta\right)\right\}} {\Gamma\left\{{\sum}_{v\neq w_{dn}}\left( l_{kv}^{(2)}+g_{kv}^{(2)}+\beta)+(l_{kw_{dn}}^{(2)}+g_{kw_{dn}}^{(2)}+\beta\right)\right\}}\\ &&\propto\frac{\Gamma\left( l_{dk;-dn}^{(1)}+1+g_{dk}^{(1)}+\alpha\right)} {\Gamma\left( l_{dk;-dn}^{(1)}+g_{dk}^{(1)}+\alpha\right)}/ \frac{\Gamma\left( l_{d.;-dn}^{(1)}+g_{d.}^{(1)}+K\alpha\right)} {\Gamma\left( l_{d.;-dn}^{(1)}+1+g_{d.}^{(1)}+K\alpha\right)}\\ &&\times\frac{\Gamma\left( l_{kw_{dn};-dn}^{(2)}+1+g_{kw_{dn}}^{(2)}+\beta\right)}{\Gamma\left( l_{kw_{dn};-dn}^{(2)}+g_{kw_{dn}}^{(2)}+\beta\right)} / \frac{\Gamma\left( l_{k.;-dn}^{(2)}+g_{k.}^{(2)}+V\beta\right)} {\Gamma\left( l_{k.;-dn}^{(2)}+1+g_{k.}^{(2)}+V\beta\right)},\\ \end{array} $$

(27)

where the subscript “ − d n” indicates counts excluding the n th word in normal document d, $l_{d\cdot }^{(1)}$ and $g_{d\cdot }^{(1)}$ are the sum of $l_{dk}^{(1)}$ and $g_{dk}^{(1)}$ over all formal topics k, and $l_{k\cdot }^{(2)}$ and $g_{k\cdot }^{(2)}$ are the sum of $l_{kv}^{(2)}$ and $g_{kv}^{(2)}$ over all words v. Noting $l_{d\cdot }^{(1)}$ is equal to the total number of words in document d and $g_{d\cdot }^{(1)}$ is equal to the total number of words in all short texts associated with normal document d, $l_{d\cdot }^{(1)}$ and $g_{d\cdot }^{(1)}$ are constant values. Using the characteristics of Γ function, which is Γ(x + 1) = xΓ(x), (27) can be simplified as

$$\begin{array}{lll} f(z_{dn}=k \mid \cdot) \propto \left( l_{dk;-dn}^{(1)}+g_{dk}^{(1)}+\alpha\right) \times \frac{l_{k,w_{dn};-dn}^{(2)}+g_{k,w_{dn}}^{(2)}+\beta}{l_{k\cdot;-dn}^{(2)}+g_{k\cdot}^{(2)}+V\beta}. \end{array} $$

For b,x,y, we can derive their conditional posterior distributions from (26) similarly.