Data Removal from an AUC Optimization Model

Abstract

Our learned model may be required to make some dynamic adjustments owing to data removals in privacy, adversarial learning, etc. Previous studies on this issue mostly focus on the standard classification accuracy. This work takes one step on data removal for AUC optimization, where previous methods can not be applied directly since AUC is measured by a sum of losses defined over pairs of instances from different classes. We develop the Data Removal algorithm for AUC optimization (DRAUC), and the basic idea is to adjust the trained model according to the removed data, rather than retrain another model again from the scratch. Our algorithm only needs to maintain some data statistics, without storing the training data in memory. For high-dimensional data, we utilize the frequent direction algorithm to approximate the second-order statistics, and solve the numerical solution based on gradient descent so as to avoid calculating the inverse of Hessian matrix. We verify the effectiveness of the proposed DRAUC both theoretically and empirically.

A Analysis of DRAUC with High-Dimensional Data

We introduce the following lemma [2] for strongly convex and smooth function:

Lemma 1

Let $\mathbf {w}^* = \arg \min _{\mathbf {w}} \mathcal {L}(\mathbf {w})$ w.r.t. $\mu$-strongly convex and $\beta$-smooth function $\mathcal {L}(\mathbf {w})$. After t iterations of gradient descent with step size $\eta _t = {2}/(\beta +\mu )$, we have

$$ \Vert \mathbf {w}_t - \mathbf {w}^* \Vert \le \left( \frac{\beta - \mu }{\beta + \mu } \right) ^t \Vert \mathbf {w}_0 - \mathbf {w}^*||. $$

We introduce a lemma for AUC optimization as follows:

Lemma 2

For bounded space $\mathcal {W}=\{\mathbf {w}:\Vert \mathbf {w}\Vert \le B\}$, let $\mathbf {w}^*=\arg \min _{\mathbf {w}\in \mathcal {W}} \mathcal {L}(\mathbf {w};S_n)$ and $\mathbf {w}^*_{-R}=\arg \min _{\mathbf {w}\in \mathcal {W}}\mathcal {L}(\mathbf {w};S_n {\setminus } R)$. For regularization parameter $\lambda >0$, we have

$$ \Vert \mathbf {w}^* -\mathbf {w}^*_{-R} \Vert \le 4B \frac{4 + \lambda }{\lambda }\left( \frac{r_+}{n_+}+\frac{r_-}{n_-}-\frac{r_+r_-}{n_+n_-}\right) . $$

Proof

From the definition of $\mathcal {L}(\mathbf {w};S_n)$, we have

$$ \Vert \nabla \mathcal {L}(\mathbf {w}_1;S_n) - \nabla \mathcal {L}(\mathbf {w}_2;S_n) \Vert \le (4+\lambda ) \Vert \mathbf {w}_1- \mathbf {w}_2\Vert , $$

where $\mathbf {w}_1, \mathbf {w}_2\in \mathcal {W}$, $\Vert \mathbf {x}_i\Vert \le 1$ and $\Vert \mathbf {x}_j\Vert \le 1$, and thus $\mathcal {L}(\mathbf {w};S_n)$ is $(4+ \lambda )$-smooth. From Cauchy’s mean-value theorem, we have

$$\begin{aligned} \left| \mathcal {L}(\mathbf {w}_1;S_n) - \mathcal {L}(\mathbf {w}_2;S_n) \right|= & {} | \nabla \mathcal {L}^\top (\kappa \mathbf {w}_1 + (1-\kappa )\mathbf {w}_2 ;S_n) (\mathbf {w}_1 - \mathbf {w}_2) | \nonumber \\\le & {} \Vert \nabla \mathcal {L}(\kappa \mathbf {w}_1 + (1-\kappa )\mathbf {w}_2 ;S_n) \Vert \Vert \mathbf {w}_1 - \mathbf {w}_2 \Vert \end{aligned}$$

(12)

where $\kappa \in [0,1]$ and $\kappa \mathbf {w}_1 + (1-\kappa )\mathbf {w}_2 \in \mathcal {W}$. We also have $\nabla \mathcal {L}(\mathbf {w}^*;S_n) = 0$, and it holds that

$$\begin{aligned} \Vert \nabla \mathcal {L}(t \mathbf {w}_1 + (1-t)\mathbf {w}_2 ;S_n) \Vert \le \max _{\mathbf {w}\in \mathcal {W}} \Vert \nabla \mathcal {L}(\mathbf {w};S_n) - \nabla \mathcal {L}(\mathbf {w}^*;S_n) \Vert \le 2B(4+\lambda ) \end{aligned}$$

which yields that $\mathcal {L}(\mathbf {w};S_n)$ is $2B(4+\lambda )$-Lipschitz from Eq. (12). Recall that $\varDelta (\mathbf {w}) = n_+ n_- \mathcal {L}(\mathbf {w}; S_n) - (n_+-r_+)(n_--r_-)\mathcal {L}(\mathbf {w};S_n {\setminus } R )$, and we have

(13)

where the first inequality holds from the optimal solution of $\mathcal {L}(\mathbf {w}^*_{-R};S_n {\setminus } R)$, and the last inequality follows from the $2B(n_+ n_- - (n_+ - r_+)(n_--r_-))(\lambda +4)$-Lipschitzness of $\varDelta (\mathbf {w})$. For $\lambda$-strongly convex function $\mathcal {L}(\mathbf {w};S_n)$, we have

$$\begin{aligned} \mathcal {L}(\mathbf {w}^*_{-R};S_n) - \mathcal {L}(\mathbf {w}^*;S_n) \ge \frac{\lambda }{2} \Vert \mathbf {w}^*_{-R} - \mathbf {w}^*\Vert ^2 \end{aligned}$$

(14)

from $\nabla \mathcal {L}(\mathbf {w}^*;S_n)=0$. Combining Eq. (13) and  (14) completes the proof.

It is necessary to introduce the following lemma from [8]:

Lemma 3

Let Z be the sketch matrix of X using frequent direction. We have

$$ \left\| X[X]^\top - Z[Z]^\top \right\| \le {2 tr( X[X]^\top )}/{m} $$

where m is the sketch size.

Proof of Theorem 2.

Proof

Let $\hat{\mathcal {L}}(\mathbf {w},S_n{\setminus } R)$ be the loss by replacing covariance matrices $\mathcal {S}^+_{-R}$ and $\mathcal {S}^-_{-R}$ with $\hat{\mathcal {S}}^+_{-R}$ and $\hat{\mathcal {S}}^-_{-R}$, and $\hat{\mathbf {w}}^*_{-R} = \mathop {\arg \min }_{\mathbf {w}} \{ \hat{\mathcal {L}}(\mathbf {w};S_n{\setminus } R) \}$, we have

$$\begin{aligned} \Vert \hat{\mathbf {w}}_{-R} - \mathbf {w}^*_{-R} \Vert \le \left\| \hat{\mathbf {w}}_{-R} - \hat{\mathbf {w}}^*_{-R} \right\| + \left\| \hat{\mathbf {w}}^*_{-R} - \mathbf {w}^*_{-R} \right\| . \end{aligned}$$

(15)

Combining with Lemma 1, this follows that

$$\begin{aligned} \Vert \hat{\mathbf {w}}_{-R} - \mathbf {w}^*_{-R} \Vert \le \left( \frac{2}{\lambda + 2} \right) ^T \left\| \mathbf {w}^* -\mathbf {w}^*_{-R} \right\| + \left( 1+\left( \frac{2}{\lambda +2} \right) ^T \right) \left\| \hat{\mathbf {w}}^*_{-R} - \mathbf {w}^*_{-R} \right\| . \end{aligned}$$

(16)

To bound $\Vert \hat{\mathbf {w}}^*_{-R} - \mathbf {w}^*_{-R}\Vert$, we first rewrite $\mathcal {L}(\mathbf {w};S_n{\setminus } R)$ as

$$ \mathcal {L}(\mathbf {w};S_n{\setminus } R) = \mathbf {w}^\top (A_1 + A_2)\mathbf {w}+ \mathbf {w}^\top \mathbf {a}+ 1/ 2 $$

where $\mathbf {a}= \mathbf {c}^-_{-R} - \mathbf {c}^+_{-R}$, $A_1 = \mathcal {S}^+_{-R} + \mathcal {S}^-_{-R} + \lambda \mathbf {I}_d$ and $A_2 = (\mathbf {c}^-_{-R} - \mathbf {c}^+_{-R})(\mathbf {c}^-_{-R} - \mathbf {c}^+_{-R})^\top$. Similarly, we rewrite $\hat{\mathcal {L}}(\mathbf {w};S_n{\setminus } R)$ as

$$ \hat{\mathcal {L}}(\mathbf {w};S_n{\setminus } R) = \mathbf {w}^\top (\hat{A}_1 + A_2)\mathbf {w}+ \mathbf {w}^\top \mathbf {a}+ 1 /2\quad \text {with}\quad \hat{A}_1 = \hat{\mathcal {S}}^+_{-R} + \hat{\mathcal {S}}^-_{-R} + \lambda \mathbf {I}_d. $$

Minimizing $\mathcal {L}(\mathbf {w};S_n)$ and $\hat{\mathcal {L}}(\mathbf {w},S_n)$ gives

$$ \mathbf {w}^*_{-R} = (A_1 + A_2)^{-1}\mathbf {a}\quad \text {and} \quad \hat{\mathbf {w}}^*_{-R} = (\hat{A}_1 + A_2)^{-1}\mathbf {a}, \quad \text {respectively}. $$

It is easy to get

$$\begin{aligned}&\left\| (A_1 + A_2)^{1/2} (\hat{A_1}+A_2)^{-1}(A_1 + A_2)^{1/2} - \mathbf {I}_d \right\| \\ =&\left\| (\hat{A}_1 + A_2)^{-1/2} (A_1-\hat{A}_1) (\hat{A}_1 + A_2)^{-1/2} \right\| \le \left\| A_1-\hat{A}_1 \right\| \left\| (\hat{A_1}+A_2)^{-1}\right\| \le \frac{2\tau }{\lambda m }, \end{aligned}$$

where $\tau = \max (rank(X^+_n[X^+_n]^\top ),rank(X^-_n[X^-_n]^\top ))$, and the inequality comes from Lemma 3. Denote $\varOmega = (A_1 + A_2)^{1/2} (\hat{A_1}+A_2)^{-1}(A_1 + A_2)^{1/2} - \mathbf {I}_d$, we have

$$\begin{aligned}&\left\| \hat{\mathbf {w}}^*_{-R} - \mathbf {w}^*_{-R} \right\| = \left\| \left( (\hat{A}_1 + A_2) ^{-1} - \left( A_1 + A_2 \right) ^{-1}\right) \mathbf {a}\right\| \nonumber \\ =&\left\| (A_1+A_2)^{-1/2} \varOmega (A_1+A_2)^{-1/2} \mathbf {a}\right\| \nonumber \le \frac{2\tau }{\lambda m } \left\| (A_1 + A_2)^{-1} \mathbf {a}\right\| \le \frac{2\tau }{\lambda m } B, \end{aligned}$$

which completes the proof by combining with Eq. (16) and Lemma 2.