Syndromic surveillance of Flu on Twitter using weakly supervised temporal topic models

Abstract

Surveillance of epidemic outbreaks and spread from social media is an important tool for governments and public health authorities. Machine learning techniques for nowcasting the Flu have made significant inroads into correlating social media trends to case counts and prevalence of epidemics in a population. There is a disconnect between data-driven methods for forecasting Flu incidence and epidemiological models that adopt a state based understanding of transitions, that can lead to sub-optimal predictions. Furthermore, models for epidemiological activity and social activity like on Twitter predict different shapes and have important differences. In this paper, we propose two temporal topic models (one unsupervised model as well as one improved weakly-supervised model) to capture hidden states of a user from his tweets and aggregate states in a geographical region for better estimation of trends. We show that our approaches help fill the gap between phenomenological methods for disease surveillance and epidemiological models. We validate our approaches by modeling the Flu using Twitter in multiple countries of South America. We demonstrate that our models can consistently outperform plain vocabulary assessment in Flu case-count predictions, and at the same time get better Flu-peak predictions than competitors. We also show that our fine-grained modeling can reconcile some contrasting behaviors between epidemiological and social models.

Appendix

1.1 HFSTM-A-FIT

In this appendix, we show the equations we designed for HFSTM-A-FIT. Note that the outlines of the HFSTM-FIT algorithm is similar to HFSTM-A-FIT, one can derive equations for HFSTM-FIT from the content we show below.

Let K, T, N, and U be the number of states, number of tweets per user, number of words per tweet, and total number of users. Let $O=<O_1,O_2,\ldots ,O_T>$ and $S=<S_1,S_2,\ldots ,S_T>$ the observed sequences of tweets and hidden states respectively for a particular user.

Here is a list of symbols that we will use.

• 1. $\epsilon$: the prior for the binary state switching variable, which determines whether state of a tweet is drawn from the transition probability matrix or simply copied from the state of the previous tweet (a number in (0, 1])

• 2. $\pi$: initial state probability (size is $1\times K$)

• 3. $\eta$: tansition probability matrix (size is $K\times K$)

• 4. $\phi$: word distrtibution for each state (size is $K\times W$, where W is the total number of keywords for all of the states)

• 5. $w_{tn}$: the nth word in the tth tweet

• 6. $\lambda$: the background switch variable

• 7. c: the topic switch variable

• 8. y: the observed aspect value

For HFSTM-A, as mentioned in Sect. 3.3, the value of $\lambda$ is biased by the observed aspect value y. We use $\lambda$ instead of $\lambda _{y}$ in the following for brevity, but remember the $\lambda$ value in the equations is actually calculated using:

$$\begin{aligned} \lambda _{y_i=0}&=\lambda \\ \lambda _{y_i=1}&=\lambda +b \,\times \,(1-\lambda )\\ c_{y_i=0}&=c-a \,\times \,c\\ c_{y_i=1}&=c+a \,\times \,(1-c) \end{aligned}$$

We want to learn all the parameters given the tweet sequence. For compact notation we use $H=(\epsilon ,\pi ,\eta ,\phi ,\lambda ,c)$. In HFSTM-A-FIT, we use forward backward procedure for which we define forward variable $A_t(i)$ and backward variable $B_t(i)$ as follows.

$$\begin{aligned} A_t(i)&= P(O_1,O_2,\ldots ,O_t, S_t =i|H)\\ B_t(i)&= P(O_{t+1},\ldots ,O_T|S_t =i,H) \end{aligned}$$

Let $\gamma _t(i)$ be the probability of being in state $S_i$ at for tth tweet given the observed tweet sequence O and other model parameters. For each user the size of $\gamma$ is $2K\times T$ (with the first K states as the states which are copies of the previous state, and the second K states which are derived after a transition). This probability can be expressed by the forward and backward probabilities.

$$\begin{aligned} \gamma _t(i)&= P(S_t =i|O,H)\\&= \frac{A_t(i)B_t(i)}{P(O|H)}\\&= \frac{A_t(i)B_t(i)}{\sum _{i=1}^{2K} A_t(i)B_t(i)}\\ \end{aligned}$$

We have two switch variables in the model: l, x. If $l=1$, the word is generated either by states or topics, if $l=0$ it’s generated by background. If $x=0$, the word is generated by topics, if $x=1$ it’s by states.

For $l_i=1$, which means that $w_i$ is generated by either state or topics.

$$\begin{aligned}&P(l_i=1| \lambda ,c,H,w) =\frac{ P(l_i=1|\lambda ,c,H)P(w|l_i=1,\lambda ,c,H)}{ P(w|\lambda ,c,H)}\\&= \textstyle \frac{ \lambda P(w_i|\lambda ,c,H,l_i=1,w_{-i})P(w_{-i}|\lambda ,c,H,l_i=1)}{ P(w_i|\lambda ,c,H,w_{-i})P(w_{-i}|\lambda ,c,H)}\\&= \textstyle \frac{ \lambda \sum _{x_i} [P(w_i|\lambda ,c,H,l_i=1,x_i,w_{-i})P(x_i|\lambda ,c,H,l_i=1,w_{-i})]}{ \sum _{l_i}[P(w_i|\lambda ,c,H,l_i,w_{-i})P(l_i|\lambda ,c,H,w_{-i})]}\\&= \textstyle \frac{ \lambda [(\sum _{topic} \phi _{topic} (w_i)P(topic|x_i=0,l_i=1,\lambda ,c,H,w_{-i}))(1-c)+(\sum _{state} \phi _{state}(w_i)\gamma _i(state))c]}{ \lambda [(\sum _{topic} \phi _{topic} (w_i)P(topic|...))(1-c)+(\sum _{state} \phi _{state}(w_i)\gamma _i(state))c]+(1-\lambda )\phi _{Bak}(w_i)} \end{aligned}$$

For $l_i=0$, $w_i$ is generated by background.

$$\begin{aligned}&P(l_i=0| \lambda ,c,H,w) =\textstyle \frac{ P(l_i=0|\lambda ,c,H)P(w|l_i=0,\lambda ,c,H)}{ P(w|\lambda ,c,H)}\\&= \textstyle \frac{ (1-\lambda )\phi _{Bak}(w_i)}{ \lambda [(\sum _{topic} \phi _{topic} (w_i)P(topic|...))(1-c)+(\sum _{state} \phi _{state}(w_i)\gamma _i(state))c]+(1-\lambda )\phi _{Bak}(w_i)} \end{aligned}$$

For $x_i=0$, $w_i$ is generated by topics.

$$\begin{aligned}&P(x_i=0| \lambda ,c,H,w) =\frac{ P(x_i=0|\lambda ,c,H)P(w|x_i=0,\lambda ,c,H)}{ P(w|\lambda ,c,H)}\\&= \frac{ (1-c) P(w_i|\lambda ,c,H,x_i=0,w_{-i})P(w_{-i}|\lambda ,c,H,x_i=0)}{ P(w_i|\lambda ,c,H,w_{-i})P(w_{-i}|\lambda ,c,H)}\\&= \frac{ (1-c) \sum _{l_i} [P(w_i|\lambda ,c,H,x_i=0,l_i,w_{-i})P(l_i|\lambda ,c,H,x_i=0,w_{-i})]}{ \sum _{x_i}P(w_i|\lambda ,c,H,w_{-i},x_i)P(x_i|\lambda ,c,H,w_{-i})}\\&=\textstyle \frac{ (1-c) [(\sum _{topic} \phi _{topic} (w_i)P(topic|x_i=0,l_i=1,\lambda ,c,H,w_{-i}))\lambda +\phi _{Bak}(w_i)(1-\lambda )]}{ (1-c) [(\sum _{top} \phi _{top} (w_i)P(top|...))\lambda +\phi _{Bak}(w_i)(1-\lambda )]+c[(\sum _{sta}\phi _{sta}(w_i)\gamma _i(sta))\lambda +\phi _{Bak}(w_i)(1-\lambda )]} \end{aligned}$$

For $x_i=1$, $w_i$ is generated by states.

$$\begin{aligned}&P(x_i=1| \lambda ,c,H,w) =\frac{ P(x_i=1|\lambda ,c,H)P(w|x_i=1,\lambda ,c,H)}{ P(w|\lambda ,c,H)}\\&= \textstyle \frac{ c[(\sum _{sta}\phi _{sta}(w_i)\gamma _i(sta))\lambda +\phi _{Bak}(w_i)(1-\lambda )]}{ (1-c) [(\sum _{top} \phi _{top} (w_i)P(top|...))\lambda +\phi _{Bak}(w_i)(1-\lambda )]+c[(\sum _{sta}\phi _{sta}(w_i)\gamma _i(sta))\lambda +\phi _{Bak}(w_i)(1-\lambda )]} \end{aligned}$$

Forward variable: We now further expand the forward variable in more details. The Initialization is as follows:

For $1\le i\le K$:

$$\begin{aligned} A_1(i)&=P(O_1,S_1=i|H)\\&=P(O_1|S_1=i,H)P((S_1=i|H)\\&=\pi _i\prod _{n=1}^NP(w_{1n}|S_1=i,H)\\&=\pi _i\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{1n})+\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{1n})P(top|\ldots )+c\phi ^i(w_{1n})\right] \right\} \end{aligned}$$

For $K+1\le i\le 2K$: $A_1(i)=0$

Induction is as follows:

For $1\le j\le K$:

$$\begin{aligned} A_t(j)&=P(O_1,O_2,\ldots ,O_t,S_t=j|H)\\&=\left( \sum _i^{2K}A_{t-1}(i)\epsilon \eta _{ij}\right) P(O_t|S_t=j,H)\\&= \left( \sum _i^{2K}A_{t-1}(i)\epsilon \eta _{ij}\right) \prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{1n})\right. \\&\left. \quad +\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{1n})P(top|\ldots )+c\phi ^j(w_{1n})\right] \right\} \end{aligned}$$

For $K+1\le j\le 2K$:

$$\begin{aligned}&A_t(j)=P(O_1,O_2,\ldots ,O_t,S_t=j|H)\\&=(A_{t-1}(j)+A_{t-1}(j-K))(1-\epsilon )\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{1n})\right. \\&\left. \quad +\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{1n})P(top|\ldots )+c\phi ^j(w_{1n})\right] \right\} \end{aligned}$$

Backward variable: The initialization for backward variable is as follows:

For $1\le i\le 2K$:

$$\begin{aligned} B_T(i)=1 \end{aligned}$$

Induction is as follows:

For $1\le i\le K$:

$$\begin{aligned}&B_t(i)=P(O_{t+1},\ldots ,O_T|S_t=i,H)\\&= \left( \sum _j^{K} \epsilon \eta _{ij} P(O_{t+1}| S_{t+1} =j,H) B_{t+1}(j)\right) \\&\quad + (1-\epsilon )P(O_{t+1}| S_{t+1} =i+K,H) B_{t+1}(i+K)\\&=\left( \sum _j^{K} \epsilon \eta _{ij} \prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})+\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )\right. \right. \right. \\&\left. \left. \left. \quad +\,c\phi ^j(w_{(t+1)n})\right] \right\} B_{t+1}(j)\right) + (1-\epsilon )\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n}) \right. \\&\left. \quad +\,\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )+c\phi ^i(w_{(t+1)n})\right] \right\} B_{t+1}(i+K)\\ \end{aligned}$$

For $K+1 \le i \le 2K$:

$$\begin{aligned}&B_{t}(i) = P(O_{t+1},\ldots ,O_T|S_t =i,H)\\&\quad = \left( \sum _j^{K} \epsilon \eta _{ij} P(O_{t+1}| S_{t+1} =j,H) B_{t+1}(j)\right) \\&\qquad + \,(1-\epsilon )P(O_{t+1}| S_{t+1} =i,H) B_{t+1}(i)\\&\begin{aligned}&\quad = \left( \sum _j^{K} \epsilon \eta _{ij} \prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})+\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )\right. \right. \right. \\&\left. \left. \left. \qquad +\,c\phi ^j(w_{(t+1)n})\right] \right\} B_{t+1}(j) \right) + (1-\epsilon )\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})\right. \end{aligned} \\&\quad \quad \left. \left. \left. +\,\lambda [(1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )+c\phi ^{i-K}(w_{(t+1)n}\right) \right] \right\} B_{t+1}(i)\\ \end{aligned}$$

Define z as follows:

$$\begin{aligned} z_{t,n}(i)&=P(T_{tn}=i|l_{tn}=1,x_{tn}=0,w_{tn},H)\\&=\frac{ P(w_{tn}|T_{tn}=i,H,l_{tn}=1,x_{tn}=0)P(T_{tn}=i|l_{tn}=1,x_{tn}=0,H)}{ P(w_{tn}|l_{tn}=1,x_{tn}=0,H)}\\&=\frac{ \phi _{top=i}(w_{tn})P(T_{tn}=i|l_{tn}=1,x_{tn}=0,H)}{ \sum _{i}[\phi _{top=i}(w_{tn})P(T_{tn}=i|l_{tn}=1,x_{tn}=0,H)]} \end{aligned}$$

Let $\xi _t(i,j)$ be the probability of being in state $S_i$ at time t, and state $S_j$ at time $t+1$, given O and other model parameters.

$$\begin{aligned} \xi _t(i,j)&=P(S_t=i,S_{t+1}=j|O,H)\\&=\frac{P(S_t=i,S_{t+1}=j,O|H)}{P(O|H)} \end{aligned}$$

To express $\xi _t(i,j)$, we have the following definition.

For $1\le i\le 2K$ and $1\le j\le K$:

$$\begin{aligned}&T_1=A_t(i)\epsilon \eta _{ij}P(O_{t+1}|S_{t+1}=j,H)B_{t+1}(j)\\&=A_t(i)\epsilon \eta _{ij}\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})\right. \\&\left. \quad +\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )+c\phi ^j(w_{(t+1)n})\right] \right\} B_{t+1}(j) \end{aligned}$$

For $1\le i\le K$ and $K+1\le j\le 2K$:

$$\begin{aligned}&T_2=A_t(i)(1-\epsilon )P(O_{t+1}|S_{t+1}=i+K,H)B_{t+1}(i+K)\\&=A_t(i)(1-\epsilon )\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})\right. \\&\left. \quad +\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )+c\phi ^i(w_{(t+1)n})\right] \right\} B_{t+1}(i+K) \end{aligned}$$

For $K+1\le i\le 2K$ and $K+1\le j\le 2K$:

$$\begin{aligned}&T_3=A_t(i)(1-\epsilon )P(O_{t+1}|S_{t+1}=i,H)B_{t+1}(i)\\&=A_t(i)(1-\epsilon )\prod _{n=1}^N\left\{ (1-\lambda )\phi _{Bak}(w_{(t+1)n})\right. \\&\left. \quad +\lambda \left[ (1-c)\sum _{top}\phi _{top}(w_{(t+1)n})P(top|\ldots )+c\phi ^{i-K}(w_{(t+1)n})\right] \right\} B_{t+1}(i) \end{aligned}$$

Correspondingly, we have the following $\xi$ values according to the different i, j value range:

$$\begin{aligned} \xi _t(i,j)&= \frac{ T_1 }{ \sum _i \sum _j (T_1 + T_2 + T_3) }\\ \xi _t(i,j)&= \frac{ T_2 }{ \sum _i \sum _j (T_1 + T_2 + T_3) }\\ \xi _t(i,j)&= \frac{ T_3 }{ \sum _i \sum _j (T_1 + T_2 + T_3) } \end{aligned}$$

Estimation of parameters:

We use the following equations to estimate the parameter values in the M-step.

For estimating $\epsilon$:

$$\begin{aligned} \epsilon&= \frac{ \sum _{u=1}^U \sum _{t=1}^T \sum _{i=1}^{2K} \sum _{j=1}^K \xi (i,j) }{ \sum _{u=1}^U \sum _{t=1}^T \sum _{i=1}^{2K} \sum _{j=1}^{2K} \xi (i,j) } \end{aligned}$$

For estimating $\pi$:

$$\begin{aligned} \pi _i&= \frac{ \sum _{u=1}^U \gamma _1(i) }{ \sum _{u=1}^U \sum _{i=1}^K \gamma _1(i) } \quad \text {for}\, 1 \le \,\hbox {i}\, \le \,\hbox {K} \end{aligned}$$

For estimating $\eta$:

$$\begin{aligned} \eta _{ij}&= \frac{ \sum _{u=1}^U \sum _{t=1}^T \left( \xi _t(i,j) + \xi _t(i+K,j)\right) }{ \sum _{u=1}^U \sum _{t=1}^T \sum _{j=1}^K \left( \xi _t(i,j) + \xi _t(i+K,j)\right) } \quad \text {for}\, 1\le \,\hbox {i} \,\le \,\hbox {K}, \,1\le \,\hbox {j}\,\le \,\hbox {K} \end{aligned}$$

For estimating $\lambda$:

$$\begin{aligned} \lambda =\frac{\sum _u\sum _t \frac{1}{N_t}\sum _{n=1}^{N_t}P(l_{tn}=1|\lambda ,c,H,w)}{UT} \end{aligned}$$

For estimating c:

$$\begin{aligned} c =\frac{\sum _u\sum _t \frac{1}{N_t}\sum _{n=1}^{N_t}P(l_{tn}=1|\lambda ,c,H,w)P(x_{tn}=1|\lambda ,c,H,w)}{\sum _u\sum _t \frac{1}{N_t}\sum _{n=1}^{N_t}P(l_{tn}=1|\lambda ,c,H,w)} \end{aligned}$$

For estimating $\phi$:

$$ \begin{aligned} \phi ^i(w)&=\textstyle \frac{\sum _{u=1}^U \sum _{t=1}^T \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=1|\lambda ,c,H,O)\left( \gamma _t(i) + \gamma _t(i+K)\right) }{\sum _{u=1}^U \sum _{t=1}^T \sum _{w=1}^W \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=1|\lambda ,c,H,O)\left( \gamma _t(i) + \gamma _t(i+K)\right) } \\&\text {for}\, 1\le \,\hbox {i}\, \le \,\hbox {K}\\ \phi _{Bak}(w)&= \frac{\sum _{u=1}^U \sum _{t=1}^T \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=0|\lambda ,c,H,O) }{\sum _{u=1}^U \sum _{t=1}^T \sum _{w=1}^W \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=0|\lambda ,c,H,O) }\\ \phi _{Topic}(w)&= \textstyle \frac{ \sum _{u=1}^U \sum _{t=1}^T \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=0|\lambda ,c,H,O)z_{t,n}(Topic) }{ \sum _{u=1}^U \sum _{t=1}^T \sum _{w=1}^W \sum _{\begin{array}{c} 1 \le n \le N \\ \& \\ w=w_{tn} \end{array}} P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=0|\lambda ,c,H,O)z_{t,n}(Topic) }\\ P(T_{tn}=i|l_{tn}&=1,x_{tn}=0,H)=\textstyle \frac{ \sum _{u=1}^U\sum _{t=1}^T\sum _{n=1}^{N_t}P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=0|\lambda ,c,H,O)z_{t,n}(i)}{ \sum _{u=1}^U\sum _{t=1}^T\sum _{n=1}^{N_t}P(l_{tn}=1|\lambda ,c,H,O)P(x_{tn}=0|\lambda ,c,H,O)} \end{aligned}$$