A word embedding-based approach to cross-lingual topic modeling

Abstract

The cross-lingual topic analysis aims at extracting latent topics from corpora of different languages. Early approaches rely on high-cost multilingual resources (e.g., a parallel corpus), which is hard to come by in many real cases. Some works only require a translation dictionary as a linkage between languages; however, when given an inappropriate dictionary (e.g., small coverage of dictionary), the cross-lingual topic model would shrink to a monolingual topic model and generate less diversified topics. Therefore, it is imperative to investigate a cross-lingual topic model requiring fewer bilingual resources. Recently, some space-mapping techniques have been proposed to help align multiple word embedding of different languages into a quality cross-lingual word embedding by referring to a small number of translation pairs. This work proposes a cross-lingual topic model, called Cb-CLTM, which incorporates with cross-lingual word embedding. To leverage the power of word semantics and the linkage between languages from the cross-lingual word embedding, the Cb-CLTM considers each word as a continuous embedding vector rather than a discrete word type. The experiments demonstrate that, when cross-lingual word space exhibits strong isomorphism, Cb-CLTM can generate more coherent topics with higher diversity and induce better representations of documents across languages for further tasks such as cross-lingual document clustering and classification. When the cross-lingual word space is less isomorphic, Cb-CLTM generates less coherent topics yet still prevails in topic diversity and document classification.

A Collapsed Gibbs Sampler for Topic Assignment

Notice that we omit $\alpha , \theta , \psi , H^{cs}, l_d$ from distribution $p(z_{d_i}=t | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}; \alpha , \theta , \psi , H^{cs}, l_d)$ and instead use $p(z_{d_i}=t | \mathbf {z}_{\lnot {d_i}}, \mathbf {w})$ for brevity, where $\mathbf {w}$ contains the words of a document.

$$\begin{aligned} p(z_{d_i}&=t | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}) \propto p(z_{d_i}=t, w_{d_i} | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}_{\lnot {d_i}} )\\&= \int {p(z_{d_i}=t, w_{d_i}, \theta _{d} | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}_{\lnot {d_i}})\text {d}\theta _{d}}\\&= \int {p(z_{d_i}=t, \theta _{d} | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}_{\lnot {d_i}})\text {d}\theta _{d}} \cdot p(w_{d_i} | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}_{\lnot {d_i}})\\&\quad \propto \underbrace{\int {p(z_{d_i}=t | \theta _{d} )p(\theta _{d} | \mathbf {z}_{\lnot {d_i}}, \mathbf {w}_{\lnot {d_i}})\text {d}\theta _{d}}}_{E(\theta _{d, t})\,of\,Dirichlet} \cdot \phi _t(w_{d_i} | \psi _{z_{d_i}=t}; H^{cs}_{{l_{d}}})\\&= \frac{(N^{t}_{\lnot {d_i}} + \alpha )}{\sum _{t=1}^{T}N_{d}^{t}+\alpha _{t}} \cdot \phi _t(w_{d_{i}} | \psi _{t};H^{cs}_{l_{d}}) \end{aligned}$$