Shrinkage Estimators for Uplift Regression

Abstract

To evaluate the efficiency of the marketing campaign or medical treatment, we divide our population into two subgroups: treatment (on which action is taken) and control (on which no action is taken), and predict the difference between effects observed in both groups. In our paper, we propose new shrinkage modifications for one of the uplift estimators called the corrected estimator. These shrinkage improvements are based on James-Stein and Ohtani approach for linear regression. In our paper, we analyze the theoretical properties of new shrinkage-corrected uplift estimators and analyze numerical results on simulated and real-life data.

Proof of Theorem 1

Proof

Notice that inequality in the thesis can be presented as:

$$\textrm{E}\left[ (\beta ^U - \hat{\beta }^U_{cJS})' X'X (\beta ^U - \hat{\beta }^U_{cJS})\right] \le \textrm{E}\left[ (\beta ^U - \hat{\beta }^U_{c})' X'X (\beta ^U - \hat{\beta }^U_{c})\right] .$$

Assuming that, $\beta ^T = \beta ^C,$ we obtain that $\beta ^U = 0.$ Then, we may reduce following expression to the following form:

$$\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' X'X {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] \le \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' X'X {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] .$$

We structure the proof as follows:

• we will show: $\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' X'X {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] \le \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right]$

• we will show: $\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] \le 4 \sigma ^2 p$

• we will show: $\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' X'X {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \ge 4 \sigma ^2 p$. This will finish the proof.

Consider left side of inequality. Assuming $W \ge X'X$ we obtain $\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' X'X {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] \le \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] .$

Now we will consider expression $\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] .$ We will show that:

(11)

Starting from the right side of the equation:

$$\begin{aligned}&\textrm{E}\left[ ({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})' W ({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})\right] - \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] + 2 \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \\ {}&= \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}- 2{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}+ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}+ 2 {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \\ {}&= \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}\right] \end{aligned}$$

We show that Eq. (11) is correct. Now we will consider parts and separately. We start with $({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})$ from

$$\begin{aligned} {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}= \left( 1- \dfrac{p-2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'(\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^{-1}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}- {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}= \dfrac{p-2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'(\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^{-1}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\end{aligned}$$

Using the fact that $\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}= 4 \sigma ^2 W^{-1},$ we may write as:

$$\begin{aligned} \begin{aligned}&\textrm{E}\left[ \left( \dfrac{p-2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'(\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^{-1}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right) 'W \left( \dfrac{p-2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'(\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^{-1}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right) \right] \\ {}&= \textrm{E}\left[ \left( \dfrac{p-2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'(\text {Var}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^{-1}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}} \right) ^2 {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \\&= \textrm{E}\left[ \left( \dfrac{(p-2)4 \sigma ^2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}} \right) ^2 {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] = \textrm{E}\left[ \dfrac{(p-2)^24^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right] \end{aligned} \end{aligned}$$

Now we will consider Notice that this is a one-dimensional expression. Using the trace operator we obtain:

$$\begin{aligned} \begin{aligned}&\textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] = \text {Tr} \left\{ \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \right\} = \textrm{E}\left[ \text {Tr} \left\{ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right\} \right] \\&= \textrm{E}\left[ \text {Tr} \left\{ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W \right\} \right] = \text {Tr} \left\{ \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W \right] \right\} = \text {Tr} \left\{ \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' \right] W \right\} \end{aligned} \end{aligned}$$

When $\beta ^T = \beta ^C$ we have $\textrm{E}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}= \beta ^U =0.$ So we may write as:

$$\begin{aligned} \begin{aligned}&\text {Tr} \left\{ \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' \right] W \right\} = \text {Tr} \left\{ \textrm{E}\left[ ({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}- \textrm{E}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}- \textrm{E}{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})' \right] W \right\} = \text {Tr} \left\{ \text {Var} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}W \right\} \\&= \text {Tr} \left\{ 4 \sigma ^2\,W^{-1} W \right\} = \text {Tr} \left\{ 4 \sigma ^2 I_p \right\} = 4 \sigma ^2 p \end{aligned} \end{aligned}$$

Now we will consider

$$\begin{aligned} \begin{aligned} \textrm{E}\left[ {{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] = \textrm{E}\left[ \sum _{i=1}^p ({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}})_i [W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}]_i \right] = \sum _{i=1}^p \textrm{E}\left[ ({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}})_i(W)_{i \boldsymbol{\cdot }} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \end{aligned} \end{aligned}$$

Firstly we need the density formula for ${{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}.$ We notice that ${{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}$ has a normal distribution with 0 mean and variance equal to $4 \sigma ^2 W^{-1}.$ Then density of ${{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}$ is given by the formula:

$$\begin{aligned} f({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}) = \dfrac{1}{(2 \pi )^{p/2}|4 \sigma ^2\,W^{-1}|^{1/2}}\text {exp} \left\{ -\frac{1}{2} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' \frac{1}{4 \sigma ^2} W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right\} . \end{aligned}$$

Now we will transform using integration by parts.

The first part is 0 because of the exponential decay of the density function of normal distribution. Now we will transform the second part. We will use the ${{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}}$ formula and calculate the derivative of multiplication.

$$\begin{aligned}&\sum _{i=1}^p \idotsint 4 \sigma ^2 \dfrac{d}{\mathop {}\!d{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_i}({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}})_i f({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}) d {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_i d {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_1 \ldots d {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_p \\&= \sum _{i=1}^p \textrm{E}4 \sigma ^2 \dfrac{d}{\mathop {}\!d{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_i}({{\,\mathrm{{\hat{\beta }^U_{cJS}}}\,}})_i =\sum _{i=1}^p \textrm{E}4 \sigma ^2 \dfrac{d}{\mathop {}\!d{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}_i}\left( 1- \dfrac{(p-2)4 \sigma ^2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) ({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})_i \\&= \sum _{i=1}^p \bigg (4\sigma ^2 \textrm{E}\left( 1- \dfrac{(p-2) 4 \sigma ^2}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) + \textrm{E}\left[ 4 \sigma ^2 \left( \dfrac{(p-2)4 \sigma ^2}{({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^2}\right) 2(W)_{i \boldsymbol{\cdot }} {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})_i \right] \bigg ) \\&=4 \sigma ^2p - \textrm{E}\left( \dfrac{p(p-2) 4^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) + \textrm{E}\left[ 4 \sigma ^2 \left( \dfrac{(p-2)4 \sigma ^2}{({{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})^2}\right) 2 {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}\right] \\&=4 \sigma ^2p - \textrm{E}\left( \dfrac{p(p-2) 4^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) + \textrm{E}\left( \dfrac{2(p-2)4^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) \\&= 4 \sigma ^2p - \textrm{E}\left( \dfrac{p(p-2) 4^2 \sigma ^4-2(p-2)4^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}'W{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) = 4 \sigma ^2p - \textrm{E}\left( \dfrac{(p-2)^24^2 \sigma ^4}{{{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}' W {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}}}\right) \end{aligned}$$

Now we back to (11) and substitute obtained results of i .

The last inequality results from the fact that matrix W is positive semi-definite. Now we will prove the last thing:

$$\textrm{E}\left[ (\beta - {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})' X'X (\beta - {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})\right] \ge 4 \sigma ^2 p.$$

Using the same line of reasoning as in and assuming that $\text {Tr} \{ W^{-1}X'X\} \ge p$ we obtain:

$$\textrm{E}\left[ (\beta - {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})' X'X (\beta - {{\,\mathrm{{\hat{\beta }^U_{c}}}\,}})\right] = \text {Tr} \left\{ 4 \sigma ^2\,W^{-1} X'X \right\} \ge 4 \sigma ^2p,$$

which ends the proof.