Robust Contextual Bandit via the Capped-$\ell _{2}$ Norm for Mobile Health Intervention

Abstract

This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in the mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expected reward, which is prone to the existence of outliers. To deal with the issue of outliers, we are the first to propose a novel robust actor-critic contextual bandit method for the mHealth intervention. In the critic updating, the capped-$\ell _{2}$ norm is used to measure the approximation error, which prevents outliers from dominating our objective. A set of weights could be achieved from the critic updating. Considering them gives a weighted objective for the actor updating. It provides the ineffective sample in the critic updating with zero weights for the actor updating. As a result, the robustness of both actor-critic updating is enhanced. There is a key parameter in the capped-$\ell _{2}$ norm. We provide a reliable method to properly set it by making use of one of the most fundamental definitions of outliers in statistics. Extensive experiment results demonstrate that our method can achieve almost identical results compared with the state-of-the-art methods on the dataset without outliers and dramatically outperform them on the datasets noised by outliers.

This work was partially supported by NSF IIS-1423056, CMMI-1434401, CNS-1405985, IIS-1718853 and the NSF CAREER grant IIS-1553687.

Appendix: The Proof of Proposition 1

Proof

The objective of (3) is non-convex and non-differentiable [2, 10]. We could obtain its sub-gradient: $\partial O\left( \mathbf {w}\right) =\sum _{i}\partial \min \left\{ \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2},\epsilon \right\} +2\zeta \mathbf {w},$ where

$$\begin{aligned} \partial \min \left\{ \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2},\epsilon \right\} ={\left\{ \begin{array}{ll} 0, &{} \text {if}\ \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}>\epsilon \\ \left[ -1,0\right] \partial \left( \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}\right) &{} \text {if}\ \ r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}=-\sqrt{\epsilon }\\ \left[ 0,1\right] \partial \left( \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}\right) &{} \text {if}\ \ r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}=\sqrt{\epsilon }\\ \partial \left( \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}\right) &{} \text {if}\ \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}<\epsilon \end{array}\right. }. \end{aligned}$$

(10)

Letting $u_{i}=1_{\left\{ \left\| r_{i}-\mathbf {x}_{i}^{T} \mathbf {w}\right\| _{2}^{2}<\epsilon \right\} }$ for $i\in \left\{ 1,\cdots ,T\right\}$ gives a simplified partial derivative of (3) that satisfies the sub-gradient (10). It is defined as

$$\begin{aligned} \partial O\left( \mathbf {w}\right) =\sum _{i}u_{i}\partial \left( \left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}\right) +2\zeta \mathbf {w}, \end{aligned}$$

which is equivalent to the partial derivative of the following objective

$$\begin{aligned} \max _{\mathbf {w}}\sum _{i}u_{i}\left\| r_{i}-\mathbf {x}_{i}^{T}\mathbf {w}\right\| _{2}^{2}+\zeta \left\| \mathbf {w}\right\| _{2}^{2}. \end{aligned}$$

(11)

From the perspective of optimization, the objective (11) is equivalent to (3).    $\square$