JSMix: a holistic algorithm for learning with label noise

Abstract

The success of deep learning is mainly dependent on large-scale and accurately labeled datasets. However, real-world datasets are marked with much noise. Directly training on datasets with label noise may lead to the overfitting. Recent research is under the spotlight on how to design algorithms that can learn robust models from noisy datasets, via designing the loss function and integrating the idea of Semi-supervised learning (SSL). This paper proposes a robust algorithm for learning with label noise that does not require additional clean data and an auxiliary model. Specifically, on the one hand, Jensen–Shannon (JS) divergence is introduced as a component of the loss function, which measures the distance between the predicted distribution and the noisy label distribution. It can alleviate the overfitting problem caused by the traditional cross entropy loss theoretically and experimentally. On the other hand, a dynamic sample selection mechanism is also proposed. The dataset is divided into the pseudo-clean labeled subset and the pseudo-noisy labeled subset. Two subsets are treated differently to exploit prior information about the data, and then learned by SSL. The dynamic sample selection is performed with the iteration between two subsets and the model parameters, which are different from the conventional training. Considering the label of the pseudo-clean labeled subset is not correct entirely, they are also refined by linear interpolation. Furthermore, we experimentally show that the integration of SSL helps the model divide two subsets more precise and build decision boundaries more explicit. Extensive experimental results on corrupted data from benchmark datasets and the real-world dataset, including CIFAR-10, CIFAR-100, and Clothing1M, demonstrate that our method is superior to many state-of-the-art approaches for learning with label noise.

Appendices

Appendices

1.1 Proof of the robustness of JS loss

Lemma 1

For any x with ground truth label $y_x$ and for any $i \not = y_x$, we have ${\mathcal {L}}_{JS}(f(x), e_i) \le \log 2$. The classifier $f(\cdot )$ contains the softmax layer.

Proof

For $f(\cdot )$ is the classifier with softmax layer, so we have $\sum \nolimits _{c=1}^C f_c(x) = 1$.

$$\begin{aligned} {\mathcal {L}}_{JS} (f(x),e_i)&= \frac{1}{2} \sum \limits _{c=1}^C \bigg (e_{ic} \log \frac{2e_{ic}}{e_{ic}+f_c(x)}) + f_c(x) \log \frac{2f_c(x)}{e_{ic}+f_c(x)} \bigg ) \\&= \frac{1}{2} \bigg ( \log \frac{2}{1+ f_i(x)} + \sum \limits _{c \not = i} f_c(x) \log 2 + f_i(x) \log \frac{2f_i(x)}{1+f_i(x)} \bigg ) \\&= \frac{1}{2} \bigg ( \log \frac{2}{1+ f_i(x)} + \sum \limits _{c \not = i} f_c(x) \log 2 + f_i(x) \log \left(2-\frac{2}{1+f_i(x)}\right)\bigg ) \\&\le \frac{1}{2} \left(\log 2 + \sum \limits _{c \not = i} f_c(x) \log 2 +f_i(x) \log 2\right) \\&= \log 2. \end{aligned}$$

Prove Finished. $\square$

Theorem 1

In a C-class classification task and any softmax output f, under symmetric noise with noise rate $\eta < 1 - 1/C$, we have

$$\begin{aligned} {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta } (f^*) - {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta } (f) < \frac{1}{C}, \end{aligned}$$

(15)

where $f^*$ is the global minimizer of ${\mathcal {R}}_{{\mathcal {L}}_{JS}}(f)$.

Proof

For any f,

$$\begin{aligned}{\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta } &= {\mathbb {E}}_{x,{\hat{y}}_x} {\mathcal {L}}_{JS}(f(x), {\hat{y}}_x) \\&= {\mathbb {E}}_{x} {\mathbb {E}}_{y_x|x} {\mathbb {E}}_{{\hat{y}}_x|x,y_x} {\mathcal {L}}_{JS}(f(x), {\hat{y}}_x) \\&= {\mathbb {E}}_{x} {\mathbb {E}}_{y_x|x} \bigg [ (1-\eta ) {\mathcal {L}}_{JS}(f(x),y_x)+\frac{\eta }{C-1} \sum \limits _{c \not =y_x}{\mathcal {L}}_{JS}({f(x), e_c})\bigg ] \\&=(1-\eta ) {\mathcal {R}}_{{\mathcal {L}}_{JS}}(f(x)) + \frac{\eta }{C-1} \bigg (\sum \limits _{c=1}^C {\mathcal {L}}_{JS}(f(x),e_c)-{\mathcal {L}}_{JS}(f(x),y_x) \bigg ) \\&= \left(1-\frac{\eta c}{C-1}) {\mathcal {R}}_{{\mathcal {L}}_{JS}}(f(x)\right) + \frac{\eta }{C-1} \sum \limits _{c=1}^C {\mathcal {L}}_{JS}(f(x),e_c). \end{aligned}$$

Since $f^*$ is the global minimizer of ${\mathcal {R}}_{{\mathcal {L}}_{JS}}(f)$ and $\eta < 1 - 1/C$,

$$\begin{aligned}&{\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f^*) -{\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f) \le (1-\frac{\eta c}{C-1})({\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f^*)-{\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f))\nonumber \\&+ \frac{\eta c}{C-1} < \frac{1}{C}. \end{aligned}$$

Prove Finished. $\square$

Theorem 2

In a C-class classification task under asymmetric noise when noise rate $\eta _{y_xc}<1-\eta _{y_x}$ with $\sum _{c \not = y_x} \eta _{y_xc}= \eta _{y_x}$, for any softmax output f, if ${\mathcal {R}}_{{\mathcal {L}}_{JS}}(f^*) =0$ we have

$$\begin{aligned} {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta } (f^*) - {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta } (f) \le B, \end{aligned}$$

(16)

where $B=C{\mathbb {E}}(1-\eta _{y_x}) \ge 0$, and $f^*$ is the global minimizer of ${\mathcal {R}}_{{\mathcal {L}}_{JS}}(f)$.

Proof

$$\begin{aligned}&{\mathcal {R}}_{JS}^{\eta }(f) = {\mathbb {E}}_{x} {\mathbb {E}}_{y_x|x} \left[(1-\eta _{y_x}){\mathcal {L}}_{JS}(f(x),y_x)\right] \\&+ {\mathbb {E}}_{x} {\mathbb {E}}_{y_x|x}\left[\sum \limits _{c \not = y_x} \eta _{y_xc}{\mathcal {L}}_{JS}(f(x),e_c) \right] \\&= {\mathbb {E}}_D \left[(1-\eta _{y_x}) ( \sum \limits _{c=1}^C {\mathcal {L}}_{JS}(f(x),e_c)-\sum \limits _{c \not = y_x} {\mathcal {L}}_{JS}(f(x),e_c)\right]\\&+ {\mathbb {E}}_D \left[ \sum \limits _{c \not = y_x} \eta _{y_xc} {\mathcal {L}}_{JS}(f(x),e_c) \right] \\&\le {\mathbb {E}}_D \left[(1-\eta _{y_x}) ( C-\sum \limits _{c \not = y_x} {\mathcal {L}}_{JS}(f(x),e_c)\right]\\&+ {\mathbb {E}}_D \left[ \sum \limits _{c \not = y_x} \eta _{y_xc} {\mathcal {L}}_{JS}(f(x),e_c) \right] \\&= C {\mathbb {E}}_D (1- \eta _{y_x}) - {\mathbb {E}}_D\left[\sum \limits _{c \not = y_x}(1-\eta _{y_x}-\eta _{y_xc}){\mathcal {L}}_{JS}(f(x),e_c) \right]. \end{aligned}$$

Thus,

$$\begin{aligned} \begin{array}{rl} &{\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f^*) - {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f) \le C {\mathbb {E}}_D(1-\eta _{y_x}) - {\mathbb {E}}_D \left[\sum \limits _{c \not = y_x} (1-\eta _{y_x}-\eta _{y_xc}){\mathcal {L}}_{JS}(f^*(x),e_c)\right] \\ &\quad + {\mathbb {E}}_D\left[\sum \limits _{c \not = y_x} (1-\eta _{y_x}-\eta _{y_xc}){\mathcal {L}}_{JS}(f(x),e_c)\right] \\ &= C {\mathbb {E}}_D(1-\eta _{y_x}) \\ &\quad + {\mathbb {E}}_D \left[\sum \limits _{c \not = y_x} (1-\eta _{y_x}-\eta _{y_xc})({\mathcal {L}}_{JS}(f(x),e_c)-{\mathcal {L}}_{JS}(f^*(x),e_c))\right]. \end{array} \end{aligned}$$

From our assumption that ${\mathcal {R}}_{{\mathcal {L}}_{JS}}(f^*) =0$, we have ${\mathcal {L}}_{JS}(f^*(x),y_x)=0$. For any $i \not = y_x$, ${\mathcal {L}}_{JS}(f^*(x),e_i)= \log 2$. And from Lemma 1, we have: ${\mathcal {L}}_{JS}(f(x), e_i) \le \log 2$ for $i \not = y_x$. So we have:

$$\begin{aligned} {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f^*) - {\mathcal {R}}_{{\mathcal {L}}_{JS}}^{\eta }(f) \le C {\mathbb {E}}_D(1-\eta _{y_x}) \triangleq B, \end{aligned}$$

where $B=C{\mathbb {E}}(1-\eta _{y_x}) \ge 0$. Prove Finished. $\square$

1.2 The value of hyperparameter $\beta$

Noise type	Symmetric noise				Asymmetric noise
Datasets	CIFAR-10		CIFAR-100		CIFAR-10/CIFAR-100/Clothing1M
Noise Rate	0.2/0.4	0.6/0.8	0.2/0.4	0.6/0.8	0.1/0.2/0.3/0.4
$\beta$	0	25	25	125	0