Contextual bandits with hidden contexts: a focused data capture from social media streams

Abstract

This paper addresses the problem of real time data capture from social media. Due to different limitations, it is not possible to collect all the data produced by social networks such as Twitter. Therefore, to be able to gather enough relevant information related to a predefined need, it is necessary to focus on a subset of the information sources. In this work, we focus on user-centered data capture and consider each account of a social network as a source that can be followed at each iteration of a data capture process. This process, whose aim is to maximize the cumulative utility of the captured information for the specified need, is constrained at each time step by the number of users that can be monitored simultaneously. The problem of selecting a subset of accounts to listen to over time is a sequential decision problem under constraints, which we formalize as a bandit problem with multiple selections. In this work, we propose a contextual UCB-like approach, that uses the activity of any user during the current step to predict his future behavior. Besides the capture of usefulness variations, considering contexts also enables to improve the efficiency of the process by leveraging some structure in the search space. However, existing contextual bandit approaches do not fit for our setting where most of the contexts are hidden from the agent. We therefore propose a new algorithm, called HiddenLinUCB, which aims at dealing with such missing information via variational inference. Experiments demonstrate the very good behavior of this approach compared to existing methods for tasks of data capture from social networks.

Appendix

1.1 Proof for Proposition 3

Following the variational principle described above, the optimal variational distribution $q_{\beta }^{\star }(\beta )$ for $\beta$ satisfies:

$$\begin{aligned} \log q_{\beta }^{\star }(\beta )&=\,\mathbb {E}_{\beta ^{\backslash }}\Bigg [ \sum \limits _{i=1}^K\sum \limits _{s\in \mathcal {A}_{i,t-1}\cup \mathcal {B}_{i,t-1}} \log p(r_{i,s} | \beta , x_{i,s}) + \log p(\beta ) \Bigg ]+C\\&=\mathbb {E}_{\beta ^{\backslash }} \Bigg [ - \dfrac{1}{2} \sum \limits _{i=1}^{K} \sum \limits _{s\in \mathcal {A}_{i,t-1}\cup \mathcal {B}_{i,t-1}} (r_{i,s}-x_{i,s}^{\top }\beta )^{2} - \dfrac{1}{2} \beta ^{\top }\beta \Bigg ]+ C \\&= \mathbb {E}_{\beta ^{\backslash }} \Bigg [ - \dfrac{1}{2} \beta ^{\top }\left( I+\sum \limits _{i=1}^{K} \sum \limits _{s\in \mathcal {A}_{i,t-1}\cup \mathcal {B}_{i,t-1}}x_{i,s}x_{i,s}^{\top } \right) \beta \\&\quad + \beta ^{\top }\left( \sum \limits _{i=1}^{K} \sum \limits _{s\in \mathcal {A}_{i,t-1}\cup \mathcal {B}_{i,t-1}}x_{i,s}r_{i,s} \right) \Bigg ] +C \end{aligned}$$

where $\mathbb {E}_{\beta ^{\backslash }}$ stands for the expectation over all terms but $\beta$ and C denotes terms which do not depend on $\beta$.

By considering the independence of factors as defined in the formulation (18), and the definitions of $V_{t-1}$ and $\hat{\beta }_{t-1}$ in the proposition, we get:

$$\begin{aligned} \log q_{\beta }^{\star }(\beta )&= -\dfrac{1}{2}(\beta -\hat{\beta }_{t-1})^{\top }V_{t-1}(\beta -\hat{\beta }_{t-1}) +C \end{aligned}$$

which corresponds to a Gaussian $\mathcal {N}(\hat{\beta }_{t-1},V_{t-1}^{-1})$.

1.2 Proof for Proposition  4

Following the variational principle described above, the optimal variational distribution $q_{x_{i,s}}^{\star }(x_{i,s})$ for every action i and step $s<t$ satisfies:

$$\begin{aligned} \log q_{x_{i,s}}^{\star }(x_{i,s})&=\mathbb {E}_{x_{i,s}^{\backslash }}[\log p(r_{i,s} | \beta ,x_{i,s})+\log p(x_{i,s}|\mu _{i},\tau _{i})]+C\\&=-\dfrac{1}{2}\mathbb {E}_{x_{i,s}^{\backslash }}\left[ (r_{i,s}-x_{i,s}^{\top }\beta )^{2} +\tau _{i}(x_{i,s}-\mu _{i})^{\top }(x_{i,s}-\mu _{i})\right] +C\\&=-\dfrac{1}{2}\mathbb {E}_{x_{i,s}^{\backslash }}\left[ x_{i,s}^{\top }(\beta \beta ^{\top }+\tau _{i}I)x_{i,s}-2x_{i,s}^{\top }(\beta r_{i,s}+\mu _{i}\tau _{i})\right] +C\\ \end{aligned}$$

where $\mathbb {E}_{x_{i,s}^{\backslash }}$ stands for the expectation over all terms but $x_{i,s}$ and C denotes terms which do not depend on $x_{i,s}$.

By considering the independence of factors as defined in the formulation (18), and the definitions of $W_{i,s}^{-1}$ and $\hat{x}_{i,s}$ in the proposition, we get:

$$\begin{aligned} \log q_{x_{i,s}}^{\star }(x_{i,s})&=-\dfrac{1}{2}(x_{i,s}-\hat{x}_{i,s})^{\top }W_{i,s}(x_{i,s}-\hat{x}_{i,s}) +C \end{aligned}$$

which corresponds to a Gaussian $\mathcal {N}(\hat{x}_{i,s},W_{i,s}^{-1})$.

1.3 Proof for Proposition  5

Following the variational principle described above, the optimal variational distribution $q_{\mu _{i}}^{\star }(\mu _{i})$ for every action i at step t satisfies:

$$\begin{aligned} \log q_{\mu _{i}}^{\star }(\mu _{i})&=\mathbb {E}_{\mu _{i}^{\backslash }}\Bigg [\log p((x_{i,s})_{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}}|\mu _{i},\tau _{i}) +\log p(\mu _{i}|\tau _{i}) \Bigg ]+C\\&=\,\mathbb {E}_{\mu _{i}^{\backslash }} \Bigg [- \dfrac{\tau _{i}}{2} \left( \sum \limits _{\begin{array}{c} s\in \mathcal {B}_{i,t-1} \\ \cup \mathcal {C}_{i,t-1} \end{array}}(x_{i,s}-\mu _{i})^{\top }(x_{i,s}-\mu _{i}) + \mu _{i}^{\top }\mu _{i} \right) \Bigg ] +C\\&=\mathbb {E}_{\mu _{i}^{\backslash }}\Bigg [- \dfrac{\tau _i}{2} \mu _{i}^{\top }\mu _{i}(1+n_{i,t-1}) + \tau _i \mu _{i}^{\top } \sum \limits _{\begin{array}{c} s\in \mathcal {B}_{i,t-1} \\ \cup \mathcal {C}_{i,t-1} \end{array}}x_{i,s} \Bigg ]+C \end{aligned}$$

where $\mathbb {E}_{\mu _{i}^{\backslash }}$ stands for the expectation over all terms but $\mu _{i}$, C denotes terms which do not depend on $\mu _{i}$ and $n_{i,t-1}=|\mathcal {B}_{i,t-1}|+|\mathcal {C}_{i,t-1}|$.

By considering the independence of factors as defined in the formulation (18), and the definitions of $\Sigma _{i,t-1}^{-1}$ and $\hat{\mu }_{i,t-1}$ in the proposition, we get:

$$\begin{aligned} \log q_{\mu _{i}}^{\star }(\mu _{i})&= -\dfrac{1}{2}(\mu _{i}-\hat{\mu }_{i,t-1})^{\top }\Sigma _{i,t-1}(\mu _{i}-\hat{\mu }_{i,t-1}) +C \end{aligned}$$

which corresponds to a Gaussian $\mathcal {N}(\hat{\mu }_{i,t-1},\Sigma _{i,t-1}^{-1})$.

1.4 Proof for Proposition  6

Following the variational principle described above, the optimal variational distribution $q_{\tau _{i}}^{\star }(\tau _{i})$ for every action i at step t satisfies:

$$\begin{aligned} \log q_{\tau _{i}}^{\star }(\tau _{i})&=\mathbb {E}_{\tau _{i}^{\backslash }} \Bigg [ \log p((x_{i,s})_{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}}|\mu _{i},\tau _{i})+ \log p(\mu _{i}|\tau _{i})+\log p(\tau _{i}) \Bigg ] +C \end{aligned}$$

where $\mathbb {E}_{\tau _{i}^{\backslash }}$ stands for the expectation over all terms but $\tau _{i}$ and C denotes terms which do not depend on $\tau _{i}$.

Let us remind that $p(\tau _{i})$ corresponds to a Gamma distribution with parameters $a_{0}$ and $b_{0}$, whose density is given by: $p(\tau _{i})=\frac{b_{0}^{b_{0}}\tau _{i}^{a_{0}-1}e^{-\tau _{i}b_{0}}}{\Gamma (a_{0})}$, where $\Gamma (a_{0})$ stands for a normalization constant. Also, since the density of a Gaussian of mean m and covariance matrix $\tau ^{-1} I$ is given by $f(x;m,\tau ^{-1} I)=\frac{\tau _{i}^{d/2}}{(2\pi )^{d/2}}e^{-\tau _{i}/2(x-m)^{\top }(x-m)}$, we get:

$$\begin{aligned} \log q_{\tau _{i}}^{\star }(\tau _{i})&= \mathbb {E}_{\tau _{i}^{\backslash }} \Bigg [ -\dfrac{1}{2}\sum \limits _{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}}\tau _{i}(x_{i,s}-\mu _{i})^{\top }(x_{i,s}-\mu _{i}) \\&\quad -\dfrac{1}{2}\tau _{i}\mu _{i}^{\top }\mu _{i} +\sum \limits _{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}} \dfrac{d}{2}\log \tau _{i} \\&\quad + \dfrac{d}{2}\log \tau _{i} +(a_{0}-1)\log \tau _{i} -b_{0}\tau _{i} \Bigg ] +C \\&= \mathbb {E}_{\tau _{i}^{\backslash }} \Bigg [ -\dfrac{\tau _{i}}{2} (n_{i,t-1}+1) \mu _{i}^{\top }\mu _{i} + \tau _{i} \mu _{i}^{\top }\sum \limits _{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}}x_{i,s} \\&\quad + -\dfrac{\tau _{i}}{2} \sum \limits _{s\in \mathcal {B}_{i,t-1} \cup \mathcal {C}_{i,t-1}}x_{i,s}^{\top }x_{i,s} \\&\quad + (a_{0}-1)\log \tau _{i} -b_{0}\tau _{i}+\dfrac{d(n_{i,t-1}+1)}{2} \log \tau _{i} \Bigg ] + C \end{aligned}$$

By considering the independence of factors as defined in the formulation (18), and the definitions of $a_{i,t-1}$ and $b_{i,t-1}$ given in the proposition, we get:

$$\begin{aligned} \log q_{\tau _{i}}^{\star }(\tau _{i})&= (a_{i,t-1}-1 )\log \tau _{i}-b_{i,t-1}\tau _{i}+C \end{aligned}$$

which corresponds to a Gamma with parameters $a_{i,t-1}$, $b_{i,t-1}$.

1.5 Proof for Proposition  8

From Proposition 3, at step t, $\beta$ follows a Gaussian distribution with mean $\hat{\beta }_{t-1}$ and variance $V_{t-1}^{-1}$. Therefore, $\beta ^{\top }\hat{\mu }_{i,t-1}$ follows a Gaussian distribution with mean $\hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1}$ and variance $\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}$, hence we directly have:  $\mathbb {P}\Big (|\beta ^{\top }\hat{\mu }_{i,t-1}-\hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1} | \le \alpha _{1} \sqrt{\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}}\Big )= 1-\delta _{1}$.

1.6 Proof for Proposition  9

From Proposition 5, at any step t, $\mu _{i}$ follows a Gaussian law of mean $\hat{\mu }_{i,t-1}$ and variance $\Sigma _{i,t-1}^{-1}$. Thus, the random variable $\mu _{i}-\hat{\mu }_{i,t-1}$ follows a Gaussian law of null mean and variance $\Sigma _{i,t-1}^{-1}$. Also, we know from this proposition that $\Sigma _{i,t-1}^{-1}=((1+n_{i,t-1})E[\tau _{i}])^{-1}I$, and thus $\Sigma _{i,t-1}^{-1}$ is diagonal with equal components on the diagonal. Let us temporarily denote $\Sigma _{i,t-1}^{-1}$ as $\sigma _{i,t}^{2} I$. Then, each component of the vector $\dfrac{\mu _{i}-\hat{\mu }_{i,t-1}}{\sigma _{i,t}}$ follows a standard Gaussian $\mathcal {N}(0,1)$, and therefore $\dfrac{||\mu _{i}-\hat{\mu }_{i,t-1}||^{2}}{\sigma _{i,t}^2}$ follows a $\chi ^{2}$ law with d freedom degrees. Let us note $\Psi$ the cumulative distribution function of the $\chi ^{2}$ law with d freedom degrees. We have:

$$\begin{aligned} \mathbb {P}\left( \dfrac{||\mu _{i}-\hat{\mu }_{i,t-1}||^{2}}{\sigma _{i,t}^{2}} \le \eta \right) = \Psi (\eta ) \end{aligned}$$

Equivalently, we have:

$$\begin{aligned} \mathbb {P}\left( ||\mu _{i}-\hat{\mu }_{i,t-1}|| \le \sqrt{\eta }\sigma _{i,t} \right) = \Psi (\eta ) \end{aligned}$$

Now let us consider that $|\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1}) | \le S||\mu _{i}-\hat{\mu }_{i,t-1}||$, since S is an upper-bound for $||\beta ||$. Therefore, we get:

$$\begin{aligned} \mathbb {P}\left( |\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1}) | \le S\sqrt{\eta }\sigma _{i,t} \right) \ge \Psi (\eta ) \end{aligned}$$

If we set $\Psi (\eta )=1-\delta _{2}$, it becomes:

$$\begin{aligned} \mathbb {P}\left( |\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1}) | \le S\sigma _{i,t}\sqrt{\Psi ^{-1}(1-\delta _{2})} \right) \ge 1-\delta _{2} \end{aligned}$$

where $\Psi ^{-1}$ stands for the inverse cumulative distribution function of a $\chi ^{2}$ law with d freedom degrees. Also, from Proposition 6, we know that $\sigma _{i,t}=\sqrt{\dfrac{1}{(1+n_{i,t-1})E[\tau _{i}]}}=\sqrt{\dfrac{b_{i,t-1}}{a_{i,t-1}(1+n_{i,t-1})}}$, which proves the proposition.

1.7 Proof for Theorem 2

From Propositions 8 and 9 and the Boole inequality, we get:

$$\begin{aligned}&P\left( |\mathbb {E}[r_{i,t}]-\hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1}|> \alpha _{1} \sqrt{\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}} + \alpha _{2}\sqrt{\dfrac{b_{i,t-1}}{a_{i,t-1}(1+n_{i,t-1})}} \right) \\&\quad = P\left( |\beta ^{\top }\hat{\mu }_{i,t-1} +\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1})-\hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1}| \right. \\&\qquad \left.>\alpha _{1} \sqrt{\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}} + \alpha _{2}\sqrt{\dfrac{b_{i,t-1}}{a_{i,t-1}(1+n_{i,t-1})}} \right) \\&\quad \le \mathbb {P}\left( |\beta ^{\top }\hat{\mu }_{i,t-1} - \hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1} |+|\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1}) | \right. \\&\qquad \left.> \alpha _{1} \sqrt{\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}}+\alpha _{2}\sqrt{\dfrac{b_{i,t-1}}{a_{i,t-1}(1+n_{i,t-1})}}\right) \\&\quad \le \mathbb {P}\left( \left\{ |\beta ^{\top }\hat{\mu }_{i,t-1} - \hat{\beta }_{t-1}^{\top }\hat{\mu }_{i,t-1} |> \alpha _{1} \sqrt{\hat{\mu }_{i,t-1}^{\top }V_{t-1}^{-1}\hat{\mu }_{i,t-1}} \right\} \text { } \right. \\&\qquad \left. \cup \text { } \left\{ |\beta ^{\top }(\mu _{i}-\hat{\mu }_{i,t-1}) | >\alpha _{2}\sqrt{\dfrac{b_{i,t-1}}{a_{i,t-1}(1+n_{i,t-1})}}\right\} \right) \\&\quad \le \delta _{1}+\delta _{2} \end{aligned}$$

By considering the contrapositive of this inequality, one obtain the announced result.