Information geometry enhanced fuzzy deep belief networks for sentiment classification

Abstract

With the development of internet, more and more people share reviews. Efficient sentiment analysis over such reviews using deep learning techniques has become an emerging research topic, which has attracted more and more attention from the natural language processing community. However, improving performance of a deep neural network remains an open question. In this paper, we propose a sophisticated algorithm based on deep learning, fuzzy clustering and information geometry. In particular, the distribution of training samples is treated as prior knowledge and is encoded in fuzzy deep belief networks using an improved Fuzzy C-Means (FCM) clustering algorithm. We adopt information geometry to construct geodesic distance between the distributions over features for classification, improving the FCM. Based on the clustering results, we then embed the fuzzy rules learned by FCM into fuzzy deep belief networks in order to improve their performance. Finally, we evaluate our proposal using empirical data sets that are dedicated for sentiment classification. The results show that our algorithm brings out significant improvement over existing methods.

Appendix

Proof for Theorem 1 Using Central Limit Theorems, for any distribution with a sufficiently large j, we have

$$({\mu _2} - {\mu _1})/{\sigma _1}\sim N(0,\,1/j)$$

$${({\sigma _2})^2}/{({\sigma _1})^2}\sim N(1,\,1/(j - 1))$$

Then, there exists a positive function c(j), which is decreasing with zero as the limit, such that

$$p\{ |\mu | \leq c(j)\}>1 - \varepsilon$$

$$p\{ |\sigma | \leq c(j)\}>1 - \varepsilon$$

For large j, we deduce that $\begin{gathered} {d_F}(({\mu _1},{\sigma _1}),({\mu _2},{\sigma _2})) \hfill \\ =\sqrt 2 \ln \{ \bigg [F(({\mu _1},{\sigma _1}),({\mu _2},{\sigma _2}))+{({\mu _1} - {\mu _2})^2}+2(\sigma _{1}^{2}+\sigma _{2}^{2})]/4{\sigma _1}{\sigma _2}\} \hfill \\ =\sqrt 2 \ln \bigg[\frac{{\sigma _{1}^{2}\sqrt {({\mu ^2}+2{\sigma ^2})({\mu ^2}+8+O(\sigma ))} +\sigma _{1}^{2}{\mu ^2}+4\sigma _{1}^{2}(1+\sigma +O({\sigma ^2}))}}{{4{\sigma _1}{\sigma _2}}}\bigg] \hfill \\ \end{gathered}$Then,

$$\sqrt 2 \ln [{c_1}(r+o(r))+1] \leq {d_F}(({\mu _1},{\sigma _1}),({\mu _2},{\sigma _2})) \leq \sqrt 2 \ln [{c_2}(r+o(r))+1]$$

where $r=\sqrt {{\mu ^2}+{\sigma ^2}}$, c₁ and c₂ are positive constants. The results holds true.

Next, we prove the superiority of d_F compared with KLD.

The symmetric form of KLD is [56]

$KLD(({\mu _1},{\sigma _1})||({\mu _2},{\sigma _2}))=\frac{1}{2}[2\ln ({\sigma _2}/{\sigma _1})+\sigma _{1}^{2}/\sigma _{2}^{2}+{({\mu _1} - {\mu _2})^2}/\sigma _{2}^{2} - 1].$

Then for large j, we have

$$KLD(({\mu _1},{\sigma _1})||({\mu _2},{\sigma _2})) \leq o(\sqrt {{\mu ^2}+{\sigma ^2}} ).$$

Then, according to Theorem 1, with at least a probability of $1 - \varepsilon$,

$$\mathop {\lim }\limits_{{n ->\infty }} KLD(({\mu _1},{\sigma _1})||({\mu _2},{\sigma _2}))/\sqrt {{\mu ^2}+{\sigma ^2}} =0$$

which implies that KLD(.) has lower sensitivity than d_F(.).