LJST: A Semi-supervised Joint Sentiment-Topic Model for Short Texts

Abstract

Several methods on simultaneous detection of sentiment and topics have been proposed to obtain subjective information such as opinion, attitude and feelings expressed in texts. Most of the techniques fail to produce desired results for short texts. In this paper, we propose LJST, a labeled joint sentiment-topic model particularly for short texts. It uses a probabilistic framework based on latent Dirichlet allocation. LJST is semi-supervised—it predicts the sentiment values for unlabeled texts in presence of a partially labeled texts with sentiment values. To address the sparsity problem in short text, we modify LJST and introduce Bi-LJST, which uses bi-terms (all possible pairs of words in a document) in place of unigrams for learning the topics by directly generating word co-occurrence patterns in each text and expressing the topics in terms of these patterns. Specifically, we have proposed a semi-supervised approach of extracting joint sentiment-topic model for short texts by incorporating bi-terms. Extensive experiments on three real-world datasets show that our methods perform consistently better than three other baselines in terms of document-level and topic-level sentiment prediction, and topic discovery—LJST using bi-term models outperforms the best baseline by producing 12% lower RMSE for document-level sentiment prediction and 6% higher F1 score for topic-sentiment prediction.

Appendix

Detailed Formulation of Model Inference for LJST

The total probability of the model based words, topics and sentiment labels can be decomposed as follows:

$$\begin{aligned} p\left( \mathbf {w}, \mathbf {z}, \mathbf {l}, \varphi , \pi , \theta , \pmb {\alpha }, \beta , \pmb {\gamma }\right) =&\prod \limits _{j=1}^{T} \prod \limits _{k=1}^{S} p\left( \varphi _{j,k};\beta \right) \times \prod \limits _{d=1}^{D} p\left( \pi _{d,j};\pmb {\gamma }\right) p\left( \theta _{d};\pmb {\alpha }\right) \times \nonumber \\&\prod \limits _{i=1}^{N_{d}} p\left( z_{d,i}|\theta _{d}\right) p\left( l_{d,j,i}|\pi _{d,j}\right) p\left( w_{d,j,k,i}|\varphi _{z_{d,i},l_{d,j,i}}\right) . \end{aligned}$$

(14)

To use collapsed Gibbs sampling, we integrate out $\varphi$, $\pi$ and $\theta$ in Eq. (14):

$$\begin{aligned} p\left( \mathbf {w}, \mathbf {z}, \mathbf {l}, \pmb {\alpha }, \beta , \pmb {\gamma }\right) =&\int _{\varphi }\int _{\pi }\int _{\theta } p\left( \mathbf {w}, \mathbf {z}, \mathbf {l}, \varphi , \pi , \theta , \pmb {\alpha }, \beta , \pmb {\gamma }\right) \,d\varphi \,d\pi \,d\theta \nonumber \\ =&\int _{\varphi } \prod \limits _{j} \prod \limits _{k} p\left( \varphi _{j,k};\beta \right) \prod \limits _{d} \prod \limits _{i} p\left( w_{d,j,k,i}|\varphi _{z_{d,i},l_{d,j,i}}\right) \,d\varphi \ \times \nonumber \\&\int _{\pi } \prod \limits _{d} \prod \limits _{j} p\left( \pi _{d,j};\pmb {\gamma }\right) \prod \limits _{i} p\left( l_{d,j,i}|\pi _{d,j}\right) \,d\pi \ \times \nonumber \\&\int _{\theta } \prod \limits _{d} \prod \limits _{i} p\left( z_{d,i}|\theta _{d}\right) p\left( \theta _{d};\pmb {\alpha }\right) \,d\theta . \end{aligned}$$

(15)

As, $\varphi$, $\pi$ and $\theta$ are independent variables, we can split the three terms of the RHS in Eq. (15) and calculate them separately. Further, each individual terms $\varphi _{j,k}$, $\pi _{d,j}$ and $\theta _{d}$ are independent. Thus, we can interchange the product and integration in each of the three terms.

For each document d, $\theta _{d}$, we replace the term $p(\theta _{d};\pmb {\alpha })$ with corresponding Dirichlet distribution and $p(z_{d,i}|\theta _{d})$ with multinomial distribution to get:

$$\begin{aligned} \int _{\theta _{d}} p\left( \theta _{d};\pmb {\alpha }\right) \prod \limits _{i} p\left( z_{d,i}|\theta _{d}\right) \,d\theta _{d} = \int _{\theta _{d}} {\left( \frac{\varGamma \left( \sum \nolimits _{j=1}^{T} \alpha _j\right) }{\prod \nolimits _{j=1}^{T}\varGamma \left( \alpha _j\right) } \right) } \theta _{d,j}^{N_{d,j}} \prod \limits _{j} \theta _{d,j}^{\alpha _{j}-1} \,d\theta _{d}. \end{aligned}$$

(16)

Using the property of Dirichlet distribution we get:

$$\begin{aligned} \int _{\theta _{d}} {\left( \frac{\varGamma \left( \sum \nolimits _{j=1}^{T} \alpha _j + \sum \nolimits _{j=1}^{T} N_{d,j}\right) }{\prod \nolimits _{j=1}^{T}\varGamma \left( \alpha _j+N_{d,j}\right) } \right) } \prod \limits _{j} \theta _{d,j}^{N_{d,j}+\alpha _{j}-1} \,d\theta _{d}&= 1. \end{aligned}$$

This leads to :

$$\begin{aligned} p(z) =&\prod \limits _{d} p\left( z_{d}\right) \nonumber \\ =&\prod \limits _{d} \int _{\theta _{d}} p\left( \theta _{d};\pmb {\alpha }\right) \prod \limits _{i} p\left( z_{d,i}|\theta _{d}\right) \,d\theta _{d} \nonumber \\ =&{\left( \frac{\varGamma \left( \sum \nolimits _{j=1}^{T} \alpha _j\right) }{\prod \nolimits _{j=1}^{T}\varGamma \left( \alpha _j\right) } \right) }^D \cdot \prod \limits _{d} \frac{\prod \nolimits _{j}\varGamma (N_{d,j} + \alpha _{j})}{\varGamma \left( N_{d} + \sum \nolimits _{j} \alpha _{j}\right) }. \end{aligned}$$

(17)

Similarly, we calculate $p(l) = \prod \nolimits _{d} \prod \nolimits _{j} p(l_{d,j})$ using the formula below:

$$\begin{aligned}&\prod \limits _{d} \prod \limits _{j} \int _{\pi _{d,j}} p\left( \pi _{d,j};\pmb {\gamma _{d}}\right) \prod \limits _{i} p\left( l_{d,j,i}|\pi _{d,j}\right) \,d\pi _{d,j} \nonumber \\&\quad = \prod \limits _{d} \prod \limits _{j} \int _{\pi _{d,j}} \prod \limits _{k} \pi _{d,j,k}^{\gamma _{d,k}-1} {\left( \frac{\varGamma \left( \sum \nolimits _{k=1}^{S} \gamma _{d,k}\right) }{\prod \nolimits _{k=1}^{S}\varGamma (\gamma _{d,k})} \right) } \pi _{d,j,k}^{N_{d,j,k}} \,d\pi _{d,j} \nonumber \\&\quad = {\left( \frac{\varGamma \left( \sum \nolimits _{k=1}^{S}\gamma _{d,k}\right) }{\prod \nolimits _{k=1}^{S}\varGamma \left( \gamma _{d,k}\right) }\right) }^{D \times T} \cdot \prod \limits _{d} \prod \limits _{j} \frac{\prod \limits _{k} \varGamma \left( N_{d,j,k}+\gamma _{d,k}\right) }{\varGamma \left( N_{d,j} + \sum \nolimits _{k}\gamma _{d,k}\right) }. \end{aligned}$$

(18)

Finally, we obtain $p(w) = \prod \nolimits _{j} \prod \nolimits _{k} p(w_{j,k})$ by replacing $\prod \nolimits _{d=1}^{D} \prod \nolimits _{i=1}^{N_{d}}$ with $\prod \nolimits _{i=1}^{V}$ in the formula

$$\begin{aligned}&\prod \limits _{j} \prod \limits _{k} \int _{\varphi _{j,k}} p\left( \varphi _{j,k};\beta \right) \prod \limits _{d} \prod \limits _{i} p\left( w_{d,j,k,i}|\varphi _{j,k}\right) \,d\varphi _{j,k} \nonumber \\&\quad = \prod \limits _{j} \prod \limits _{k} \int _{\varphi _{j,k}} p\left( \varphi _{j,k};\beta \right) \prod \limits _{i} p\left( w_{j,k,i}|\varphi _{j,k}\right) \,d\varphi _{j,k} \nonumber \\&\quad = \prod \limits _{j} \prod \limits _{k} \int _{\varphi _{j,k}} \prod \limits _{i} \varphi _{j,k,i}^{\beta -1} {\left( \frac{\varGamma (\sum \nolimits _{i=1}^{V} \beta )}{\prod \nolimits _{i=1}^{V}\varGamma (\beta )} \right) } \varphi _{j,k,i}^{N_{j,k,i}} \,d\varphi _{j,k} \nonumber \\&\quad = {\left( \frac{\varGamma (V\beta )}{{\varGamma (\beta )}^{V}} \right) }^{T \times S} \cdot \prod \limits _{j} \prod \limits _{k} \frac{\prod \nolimits _{i} \varGamma \left( N_{j,k,i}+\beta \right) }{\varGamma \left( N_{j,k}+V \beta \right) }. \end{aligned}$$

(19)

The goal of Gibbs sampling is to calculate $p(\mathbf {z}, \mathbf {l} \,|\, w_t, \pmb {\alpha }, \beta , \pmb {\gamma })$ instead of Eq. (15), by taking approximation $p(z_t, l_t \,|\, w_t,\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \pmb {\gamma })$, for each word $w_t$, in document d. Here t is the index of the word in document d.

$$\begin{aligned}&p\left( z_{t}, l_{t} \,|\, w_t,\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \nonumber \\&\quad \propto \ p\left( z_{t}, l_{t} , \mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}} \,|\, w_t, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \nonumber \\&\quad = p\left( z_{t}, l_{t}\,|\, w_t, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \cdot p\left( , \mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}} \,|\, w_t, \pmb {\alpha }, \beta , \pmb {\gamma }\right) . \end{aligned}$$

(20)

If we put the values from Eqs. (17), (18) and  (19) in each of the terms in Eq. (20), we get

$$\begin{aligned}&p\left( z_{t}, l_{t} \,|\, w_t,\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \propto \nonumber \\&\quad \prod \limits _{j} \prod \limits _{k} \frac{\varGamma \left( N_{j,k,w_t}+\beta \right) }{\varGamma \left( N_{j,k}+V \beta \right) } \cdot \prod \limits _{j} \frac{\prod \nolimits _{k} \varGamma \left( N_{d,j,k}+\gamma _{d,k}\right) }{\varGamma \left( N_{d,j} + \sum \limits _{k}\gamma _{d,k}\right) } \cdot \frac{\prod \nolimits _{j}\varGamma \left( N_{d,j} + \alpha _{j}\right) }{\varGamma \left( N_{d} + \sum \nolimits _{j} \alpha _{j}\right) }, \end{aligned}$$

(21)

and

$$\begin{aligned}&p\left( z_{t} = j, l_{t} = k \,|\, w_t,\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \propto \nonumber \\&\quad \frac{\varGamma \left( N_{j,k,w_t}+\beta \right) }{\varGamma \left( N_{j,k}+V \beta \right) } \cdot \frac{\varGamma \left( N_{d,j,k}+\gamma _{d,k}\right) }{\varGamma \left( N_{d,j} + \sum \nolimits _{k}\gamma _{d,k}\right) } \cdot \frac{\varGamma \left( N_{d,j} + \alpha _{j}\right) }{\varGamma \left( N_{d} + \sum \nolimits _{j} \alpha _{j}\right) }. \end{aligned}$$

(22)

For further simplification, we use formula $\varGamma (x+1) = x\cdot \varGamma (x)$ to get

$$\begin{aligned} \varGamma \left( N_{j,k,w_t}+\beta \right) = \varGamma \left( N_{j,k,w_t}^{-t}+\beta +1\right) \propto N_{j,k,w_t}^{-t}+\beta . \end{aligned}$$

Following this on other terms like $N_{j,k}, N_{d,j,k}, N_{d,j}$ and $N_{d}$ we get

$$\begin{aligned}&p\left( z_t = j, l_t = k \,|\, \mathbf {w},\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \gamma \right) \propto \nonumber \\&\quad \frac{N_{j,k,w_t}^{-t} + \beta }{N_{j,k}^{-t} + V \beta } \cdot \frac{N_{d,j,k}^{-t} + \gamma _{d,k}}{N_{d,j}^{-t} + \sum \nolimits _{k}\gamma _{d,k}} \cdot \frac{N_{d,j}^{-t} + \alpha _j}{N_{d}^{-t} + \sum \nolimits _{j} \alpha _j}. \end{aligned}$$

(23)

Detailed Formulation of Model Inference for Bi-LJST

There are some changes in the probability distribution for Bi-LJST. Let $\mathcal{B}$ snd $B_d$ denote the vocabulary of bi-terms and the number of bi-terms in the document d. Further assume that a bi-term is represented as $b_t = (w_p,w_q)$. The joint probability of bi-terms, topics and sentiment labels can be captured by,

$$\begin{aligned} p\left( b_t, \mathbf {z}, \mathbf {l}\right)&= p\left( b_t \,|\, \mathbf {l}, \mathbf {z}\right) p(\mathbf {l}| \mathbf {z}) p(\mathbf {z}). \end{aligned}$$

(24)

As the terms, $p(\mathbf {l}| \mathbf {z})$ and $p(\mathbf {z})$ do not depend on bi-term $b_{t}$, we can directly make use of Eqs. (17) and (18). Further, in the generative process of Bi-LJST, word pairs within a bi-term are conditionally independent i.e. $p(b_t \,|\, \mathbf {l}, \mathbf {z}) = p(w_p \,|\, \mathbf {l}, \mathbf {z}) \cdot p(w_q \,|\, \mathbf {l}, \mathbf {z})$. This leads us to topic sentiment assignment corresponding to bi-term $b_t$ as follows:

$$\begin{aligned}&p\left( z_t = j, l_t = k \,|\, b_t,\mathbf {z}^{-\mathbf{t}}, \mathbf {l}^{-\mathbf{t}}, \pmb {\alpha }, \beta , \pmb {\gamma }\right) \propto \nonumber \\&\quad \frac{\left( N_{j,k,w_p}^{-t} + \beta \right) \cdot \left( N_{j,k,w_q}^{-t} + \beta \right) }{\left( N_{j,k}^{-t} + V \beta \right) ^2} \cdot \frac{N_{d,j,k}^{-t} + \gamma _{d,k}}{N_{d,j}^{-t} + \sum \nolimits _{k}\gamma _{d,k}} \cdot \frac{N_{d,j}^{-t} + \alpha _j}{N_{d}^{-t} + \sum \nolimits _{j} \alpha _j}, \end{aligned}$$

(25)

where, $p_1$ and $p_2$ are the index of $w_p$ and $w_q$ respectively in vocabulary.