Integrating learned and explicit document features for reputation monitoring in social media

Abstract

Currently, monitoring reputation in social media is probably one of the most lucrative applications of information retrieval methods. However, this task poses new challenges due to the dynamicity of contents and the need for early detection of topics that affect the reputations of companies. Addressing this problem with learning mechanisms that are based on training data sets is challenging, given that unseen features play a crucial role. However, learning processes are necessary to capture domain features and dependency phenomena. In this work, based on observational information theory, we define a document representation framework that enables the combination of explicit text features and supervised and unsupervised signals into a single representation model. Our theoretical analysis demonstrates that the observation information quantity (OIQ) generalizes the most popular representation methods, in addition to capturing quantitative values, which is required for integrating signals from learning processes. In other words, the OIQ allows us to give the same treatment to features that are currently managed separately. Empirically, our experiments on the reputation-monitoring scenario demonstrated that adding features progressively from supervised (in particular, Bayesian inference over annotated data) and unsupervised learning methods (in particular, proximity to clusters) increases the similarity estimation performance. This result is verified under various similarity criteria (pointwise mutual information, Jaccard and Lin’s distances and the information contrast model). According to our formal analysis, the OIQ is the first representation model that captures the informativeness (specificity) of quantitative features in the document representation.

Appendix: formal proofs

Proposition 3.1

The proof is straightforward. According to the fuzzy set operators:

$$\begin{aligned} {\mathcal {O}}_{\Gamma }(d_1)\cup {\mathcal {O}}_{\Gamma }(d_2) \cup \ldots \cup {\mathcal {O}}_{\Gamma }(d_n)=(\Gamma ,f) \ , \end{aligned}$$

where

$$\begin{aligned} f(\gamma )=\max \limits _{ 1 \le i \le n} {\gamma }(d_i), \forall \gamma \in \Gamma \ . \end{aligned}$$

$\square$

Property 3.1

From ${\gamma }(d_1) \ge {\gamma }(d_2), \ \ \forall \gamma \in \Gamma$, it follows:

$$\begin{aligned} \Big \{d \in \mathcal {D} : {\gamma }(d) \ge {\gamma }(d_1), \ \forall \gamma \in \Gamma \Big \} \subseteq \Big \{d \in \mathcal {D} :{\gamma }(d) \ge {\gamma }(d_2), \ \forall \gamma \in \Gamma \Big \} \ . \end{aligned}$$

Then,

$$\begin{aligned} P_{d \in \mathcal {D}}\big ( {\gamma }(d) \ge {\gamma }(d_1), \ \forall \gamma \in \Gamma \big ) \le P_{d \in \mathcal {D}}\big ( {\gamma }(d) \ge {\gamma }(d_2), \ \forall \gamma \in \Gamma \big ) \ . \end{aligned}$$

This implies that

$$\begin{aligned} P_{d \in \mathcal {D}}\Big ({\mathcal {O}}_{\Gamma }(d)\supseteq {\mathcal {O}}_{\Gamma }(d_1)\Big )\le P_{d \in \mathcal {D}}\Big ({\mathcal {O}}_{\Gamma }(d)\supseteq {\mathcal {O}}_{\Gamma }(d_2)\Big ). \end{aligned}$$

And therefore, according to Definition 3.3:

$$\begin{aligned} {\mathcal {I}}_{\Gamma }\big (d_1\big ) \ge {\mathcal {I}}_{\Gamma }\big (d_2\big ). \end{aligned}$$

$\square$

Property 3.2

Notice that if we add a feature, the new observation is more restrictive than the initial observation, and thus, the set of messages which verify the new observation is contained in the set of messages which verify the initial observation, ${\mathcal {O}}_{\Gamma \cup \{\gamma '\}}(d) \subseteq {\mathcal {O}}_{\Gamma }(d)$. Then,

$$\begin{aligned} P_{d' \in \mathcal {D}}\Big ({\mathcal {O}}_{\Gamma \cup \{\gamma '\}}(d')\supseteq {\mathcal {O}}_{\Gamma \cup \{\gamma '\}}(d)\Big )\le P_{d' \in \mathcal {D}}\Big ({\mathcal {O}}_{\Gamma }(d')\supseteq {\mathcal {O}}_{\Gamma }(d)\Big ). \end{aligned}$$

And therefore, according to Definition 3.3:

$$\begin{aligned} {\mathcal {I}}_{\Gamma \cup \{\gamma '\}}\big (d\big ) \ge {\mathcal {I}}_{\Gamma }\big (d\big ). \end{aligned}$$

$\square$

Property 3.3

By Proposition 3.1:

$$\begin{aligned} {\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_1)\cup {\mathcal {O}}_{\Gamma }(d_2)\big ) = - \log \bigg (P_{d \in \mathcal {D}}\Big ( {\gamma }(d) \ \ge \ \max \big \{{\gamma }(d_1),{\gamma }(d_2)\big \}, \ \forall \gamma \in \Gamma \Big ) \bigg ) \ . \end{aligned}$$

Given that

$$\begin{aligned} P_{d \in \mathcal {D}}\Big ( {\gamma }(d) \ \ge \ \max \big \{{\gamma }(d_1),{\gamma }(d_2)\big \}, \ \forall \gamma \in \Gamma \Big ) \le P_{d \in \mathcal {D}}\Big ( {\gamma }(d) \ \ge \ {\gamma }(d_1), \ \forall \gamma \in \Gamma \Big ) \ , \end{aligned}$$

we finally get, ${\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_1)\cup {\mathcal {O}}_{\Gamma }(d_2)\big )\ge {\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_1)\big )$. Similarly, we can get the same result for $d_2$. $\square$

Property 3.4

By hypothesis,

$$\begin{aligned} P_{d' \in \mathcal {D}}\big ({\gamma _1}(d') \ge {\gamma _1}(d)\big ) \le P_{d' \in \mathcal {D}}\big ({\gamma _2}(d') \ge {\gamma _2}(d) \big ) \ , \end{aligned}$$

which is equivalent to

$$\begin{aligned}&\frac{1}{P_{d' \in \mathcal {D}}\big ({\gamma _2}(d') \ge {\gamma _2}(d)\big )} \ge \frac{1}{P_{d' \in \mathcal {D}}\big ({\gamma _1}(d') \ge {\gamma _1}(d) \big )} \Rightarrow \\&\quad \log \left( \frac{1}{P_{d' \in \mathcal {D}}\big ({\gamma _2}(d') \ge {\gamma _2}(d)\big )}\right) \ge \log \left( \frac{1}{P_{d' \in \mathcal {D}}\big ({\gamma _1}(d') \ge {\gamma _1}(d) \big )}\right) \Rightarrow \\&\quad \Rightarrow {\mathcal {I}}_{\{\gamma _1\}}\big (d\big )\ge {\mathcal {I}}_{\{\gamma _2\}}\big (d\big ) \ . \end{aligned}$$

$\square$

Property 3.5

Consider two features, $\gamma _1$, $\gamma _2 \in \Gamma$, given a message, $d \in \mathcal {D}$, it produces an observation under $\gamma _1$, ${\mathcal {O}}_{\gamma _1}(d)$, whose Observation Information Quantity is:

$$\begin{aligned} {\mathcal {I}}_{\{\gamma _1\}}\big (d\big ) = P_{d' \in \mathcal {D}}\big ({\gamma _1}(d') \ge {\gamma _1}(d)\big ) = P_{d' \in \mathcal {D}}\Big ( g\big ( {\gamma _2}(d')\big ) \ge g\big ( {\gamma _2}(d)\big ) \Big ) \ . \end{aligned}$$

Given that g is a strict monotonic function,

$$\begin{aligned}&P_{d' \in \mathcal {D}}\Big ( g\big ( {\gamma _2}(d')\big ) \ge g\big ( {\gamma _2}(d)\big ) \Big ) = P_{d' \in \mathcal {D}}\Big ( g\big ( {\gamma _2}(d')\big ) \ge g\big ( {\gamma _2}(d)\big ) , {\gamma _2}(d')\ge {\gamma _2}(d)\Big ) = \\&\quad = {\mathcal {I}}_{\{\gamma _1,\gamma _2\}}\big (d\big ). \end{aligned}$$

$\square$

Property 3.6

Assume that we have a finite set of messages, the proof of this proposition is a direct consequence of the representativity of the messages by the features. If we have an infinite set of features, then they will describe every message, and messages will be unequivocally determined by the values of a set of features. $\square$

Property 3.7

Given a fixed message, $d \in \mathcal {D}$, consider all the messages, $d' \in \mathcal {D}$, which verify the inequalities:

$$\begin{aligned} {\gamma }(d') \le {\gamma }(d) \ \wedge \ {\gamma ^{-1}}(d') \le {\gamma ^{-1}}(d) \ . \end{aligned}$$

These inequalities are equivalent to (by definition of $\gamma ^{-1}$):

$$\begin{aligned} {\gamma }(d') \le {\gamma }(d) \ \wedge \ \frac{1}{{\gamma }(d')} \le \frac{1}{{\gamma }(d)} \ . \end{aligned}$$

Notice that ${\gamma }(d)$ and ${\gamma }(d')$ are non-negative numbers; therefore, these inequalities imply that: ${\gamma }(d) = {\gamma }(d')$. Then, the Observation Information Quantity is:

$$\begin{aligned} {\mathcal {I}}_{\left\{ \gamma , \gamma ^{-1}\right\} }\big (d\big )= -\log \bigg (P_{d' \in \mathcal {D}}\Big ( {\gamma }(d') \le {\gamma }(d) \wedge {\gamma ^{-1}}(d') \le {\gamma ^{-1}}(d) \Big ) \bigg ) \ \end{aligned}$$

which is equivalent to:

$$\begin{aligned} {\mathcal {I}}_{\left\{ \gamma , \gamma ^{-1}\right\} }\big (d\big )= -\log \Big (P_{d' \in \mathcal {D}}\big ( {\gamma }(d') = {\gamma }(d) \big ) \Big ) \ . \end{aligned}$$

$\square$

Proposition 5.1

Given the vocabulary, $\mathcal {V} = \{w_1, \ldots , w_n\}$, consider the set of features as $\Gamma = \{occ_{w_1},\ldots ,occ_{w_n}\}$, and given a message from the collection, $d \in \mathcal {D}$, we are interested in computing the described OIQ, ${\mathcal {I}}_{occ_{w_i}}\big (d\big )$.

Assuming information additivity and considering text words as basic linguistic units, we have

$$\begin{aligned} {\mathcal {I}}_{occ_{w_i}}\big (d\big ) = \sum _{w_j \in d} {\mathcal {I}}_{occ_{w_i}}\big (w_j\big ) = \sum _{ w_j \in d} - \log \Big ( P_{w' \in \mathcal {V}}\big ({occ_{w_i}}(w') \ge {occ_{w_i}}(w_j) \big ) \Big ) \ . \end{aligned}$$

Notice that, if $w_j \ne w_i$, then ${occ_{w_i}}(w_{j}) = 0$. Thus, $P\big ({occ_{w_i}}(w') \ge 0 \big ) = 1$, since by definition ${occ_{w_i}}(d) \ge 0$, $\forall d \in \mathcal {D}$. Therefore, in the last summation all the terms are null, except for $w_{j} = w_{i}$. In this case, we have that ${occ_{w_i}}(w_i) = 1$, and given that by definition of the function ${occ_{w_i}}(.)$, its maximum value is 1, we can say that ${occ_{w_i}}(w') \ge 1$ is equivalent to ${occ_{w_i}}(w') = 1$. Therefore, the probability $P\big ({occ_{w_i}}(w') = 1 \big )$ is exactly $P(w' = w_i) = P(w_i)$. And, ${\mathcal {I}}_{occ_{w_i}}\big (d\big ) \propto - \log \big ( P(w_i) \big )$.

One of the assumptions is that every word is equiprobable, i.e. $P(w_i) = k$, $1 \le i \le n$, for an arbitrary k. In order to achieve the result, we can choose k in such a way that $- \log (k) = 1$. And finally, the summation gives us the $tf(w_i, d)$. $\square$

Proposition 5.2

Given the vocabulary, $\mathcal {V} = \{w_1, \ldots , w_n\}$, considering the set of features as, $\Gamma = \{occ_{w_1},\ldots ,occ_{w_n}\}$, and given a message from the collection, $d \in \mathcal {D}$, we are interested in computing the described OIQ, ${\mathcal {I}}_{occ_{w_i}}\big (d\big )$.

Assuming information additivity and considering messages as basic linguistic units, we have

$$\begin{aligned} {\mathcal {I}}_{occ_{w_i}}\big (d\big ) = \sum _{w_j \in d} {\mathcal {I}}_{occ_{w_i}}\big (w_j\big ) = \sum _{ w_j \in d} - \log \Big ( P_{d' \in \mathcal {D}}\big ({occ_{w_i}}(d') \ge {occ_{w_i}}(w_j) \big ) \Big ) \ . \end{aligned}$$

Notice that, if $w_{j} \ne w_{i}$, then ${occ_{w_i}}(w_j) = 0$. Thus, $P_{d' \in \mathcal {D}}\big ({occ_{w_i}}(d') \ge 0 \big ) = 1$, since by definition ${occ_{w_i}}(d') \ge 0$, $\forall d' \in \mathcal {D}$. Therefore, in the last summation all the terms are null, except for $w_{j} = w_{i}$. In this case, we have as many terms as the number of times that the word $w_i$ appears in the message d, i.e. $tf(w_i, d)$. Moreover, we have that ${occ_{w_i}}(w_j) = 1$, and given that by definition of ${occ_{w_i}}(.)$, its maximum value is 1, we can say that ${occ_{w_i}}(d') \ge 1$ is equivalent to ${occ_{w_i}}(d') = 1$. Therefore, the expression $-\log \Big (P_{d' \in \mathcal {D}}\big ({occ_{w_i}}(d') = 1 \big ) \Big )$ is exactly $-\log \Big ( P_{d' \in \mathcal {D}}(w_i \in d')\Big ) = idf(w_i)$. And thus, ${\mathcal {I}}_{occ_{w_i}}\big (d\big ) = tf(w_i, d) \cdot idf(w_i)$. $\square$

Proposition 5.3

Given the vocabulary, $\mathcal {V} = \{w_1, \ldots , w_n\}$, and considering as features:

$$\begin{aligned} {\gamma _{i,j}}(d)= {\left\{ \begin{array}{ll} 1, &{} \quad \text {if the }j^{th}\text { element in }d\text { is the word}~~ w_i \\ 0, &{} \quad \text {otherwise}\\ \end{array}\right. }, \ \ 1 \le i, j \le m. \end{aligned}$$

Assuming feature independence, we have

$$\begin{aligned} {\mathcal {I}}_{\Gamma }\big ((w_1,\ldots ,w_m)\big ) =&\sum _{\gamma _{i,j} \in \Gamma } {\mathcal {I}}_{\{\gamma _{i,j}\}}\big ((w_1, \ldots , w_m)\big ) = \\ =&\sum _{\gamma _{i,j} \in \Gamma } -\log \Big ( P_{d\in {\mathcal {D}}}\Big ( {\gamma _{i,j}}(d)\ge {\gamma _{i,j}}(w_1, \ldots , w_m)\Big )\Big ) \ . \end{aligned}$$

Being $\mathcal {S}$ all the possible sequences of words which form a message, in the previous formula we have that:

$$\begin{aligned}&P_{d\in {\mathcal {D}}}\Big ( {\gamma _{i,j}}(d)\ge {\gamma _{i,j}}(w_1, \ldots , w_m)\Big ) = \\&\quad = P_{(w_1', \ldots ,w_k') \in \mathcal {S}}\Big ( {\gamma _{i,j}}(w_1', \ldots , w_k') \ge {\gamma _{i,j}}(w_1, \ldots , w_m) \Big ) \ . \end{aligned}$$

Notice that ${\gamma _{i,j}}(w_1, \ldots , w_m)$ is equal to zero for all the sequences of the form $(w_1, \ldots , w_m)$ except for the sequences which verify that $w_j = w_i$. Since by definition, ${\gamma _{i,j}}(.) \ge 0$, in the summation all the terms are null, except for the sequences which verify $w_j = w_i$. In these cases, we have that ${\gamma _{i,j}}(w_1, \ldots , w_m) = 1$, and given that by definition of ${\gamma _{i,j}}(.)$, its maximum value is 1, we can say that ${\gamma _{i,j}}(w_1', \ldots , w_k') \ge 1$ is equivalent to ${\gamma _{i,j}}(w_1', \ldots , w_k') = 1$. Therefore, we have the next equality on probabilities:

$$\begin{aligned} P_{(w_1', \ldots ,w_k') \in \mathcal {S}}\Big ( {\gamma _{i,j}}(w_1', \ldots , w_k') = 1 \Big ) = P_{(w_1', \ldots ,w_k') \in \mathcal {S}}\Big ( w_i' = w_i, \Big ) \ . \end{aligned}$$

And finally, with trivial algebraic operations, we have:

$$\begin{aligned} Perplexity(w_1,\ldots ,w_m)=2^{\frac{1}{m}{\mathcal {I}}_{\Gamma }\big ((w_1, \ldots , w_m)\big )} \ . \end{aligned}$$

$\square$

Proposition 5.4

Considering the definition of Lin’s distance and assuming information additivity,

$$\begin{aligned} Lin(d_1, d_2) = \frac{ \displaystyle \sum _{w\in {d_1 \cap d_2}} {\mathcal {I}}_{\Gamma }\big (w\big )}{\displaystyle \sum _{w\in d_1} {\mathcal {I}}_{\Gamma }\big (w\big )+\displaystyle \sum _{w \in d_2} {\mathcal {I}}_{\Gamma }\big (w\big )} \ . \end{aligned}$$

Assuming feature independence, it is equivalent to:

$$\begin{aligned} \frac{{\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_1) \cap {\mathcal {O}}_{\Gamma }(d_2)\big )}{{\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_1)\big ) + {\mathcal {I}}\big ({\mathcal {O}}_{\Gamma }(d_2)\big )} \ . \end{aligned}$$

$\square$

Proposition 5.5

We will start from the $IDF_{N\hbox {-}gram}$ term weighting of an n-gram, g, using the same notation of $|\phi (g)|$ for the message frequency of g, and $|\mu (g)|$ for the amount of messages containing at least one subsequence of the n-gram g.

$$\begin{aligned} IDF_{N\hbox {-}gram}(g)= & {} \log \frac{|\mathcal {D}|}{|\mu (g)|} - \log \frac{|\mu (g)|}{|\phi (g)|} = \log \frac{|\mathcal {D}|}{|\mu (g)|} - \log \frac{|\mu (g)| \cdot |\mathcal {D}|}{|\phi (g)| \cdot |\mathcal {D}|} = \\= & {} \log \frac{|\mathcal {D}|}{|\mu (g)|} - \bigg ( \log \frac{|\mathcal {D}|}{|\phi (g)|} - \log \frac{|\mathcal {D}|}{|\mu (g)|} \bigg ) = 2 \cdot \log \frac{|\mathcal {D}|}{|\mu (g)|} - \log \frac{|\mathcal {D}|}{|\phi (g)|} \ . \end{aligned}$$

Considering both sets of features, $\Gamma$ and $\Gamma '$, the OIQ of an n-gram in each set of features is computed by:

$$\begin{aligned} {\mathcal {I}}_{\Gamma }\big (g\big )=\log \frac{|\mathcal {D}|}{|\phi (g)|} \ \ \ \wedge \ \ \ {\mathcal {I}}_{\Gamma '}\big (g\big )= \log \frac{|\mathcal {D}|}{|\mu (g)|} \ . \end{aligned}$$

Replacing these expressions in the definition of the $IDF_{N\hbox {-}gram}$:

$$\begin{aligned} IDF_{N\hbox {-}gram}(g) = 2 \cdot {\mathcal {I}}_{\Gamma '}\big (g\big ) - {\mathcal {I}}_{\Gamma }\big (g\big ) \ . \end{aligned}$$

$\square$

Proposition 5.6

We will start from the $\mu _{d}(\gamma _i)$ term weighting of a message.

$$\begin{aligned} \mu _{d}(\gamma _i)= & {} \log \frac{P(\gamma _i \in d \ | \ \mathcal {D}_1)}{P(\gamma _i \in d \ | \ \mathcal {D}_2)} = \log \frac{P(\gamma _i \in d \ | \ \mathcal {D}_1) \cdot |\mathcal {D}|}{P(\gamma _i \in d \ | \ \mathcal {D}_2) \cdot |\mathcal {D}|} = \\= & {} \log \frac{|\mathcal {D}|}{P(\gamma _i \in d \ | \ \mathcal {D}_2)} - \log \frac{|\mathcal {D}|}{P(\gamma _i \in d \ | \ \mathcal {D}_1)} \ . \end{aligned}$$

Considering two different scenarios relating to the relevancy of messages, $\mathcal {D}_1$ and, $\mathcal {D}_2$, the OIQ of a message is computed by:

$$\begin{aligned} {\mathcal {I}}_{\{\gamma _i\}}^{{\mathcal {D}}_1}\big (d\big )= \log \frac{|\mathcal {D}|}{P(\gamma _i \in d \ | \ \mathcal {D}_1)} \ \ \ \wedge \ \ \ {\mathcal {I}}_{\{\gamma _i\}}^{{\mathcal {D}}_2}\big (d\big )= \log \frac{|\mathcal {D}|}{P(\gamma _i \in d \ | \ \mathcal {D}_2)} \ . \end{aligned}$$

Replacing these expressions, we get:

$$\begin{aligned} \mu _{d}(\gamma _i) = {\mathcal {I}}_{\{\gamma _i\}}^{{\mathcal {D}}_2}\big (d\big ) - {\mathcal {I}}_{\{\gamma _i\}}^{{\mathcal {D}}_1}\big (d\big ) \ . \end{aligned}$$

$\square$