Towards an improved label noise proportion estimation in small data: a Bayesian approach

Abstract

Today’s classification task is getting more and more complex. This inevitably renders unanticipated compromises on the quality of data labels. In this paper, we consider learning label noise robust classifiers with focus on the tasks with limited training examples relative to the number of data classes and data dimensionality. In such cases, the existing label noise models tend to inaccurately estimate the noise proportions leading to suboptimal performance. To alleviate the problem, we formulated a regularised label noise model capable of expressing preference on the noise parameters. In addition, we treated the regularisation from a Bayesian perspective so that the regularisation parameters can be inferred from the data through the noise model, thereby facilitating model selection in the presence of label noise. This results in a more data and computationally efficient Bayesian label noise model which could be incorporated into any probabilistic classifier, including those that are known to be data intensive such as deep neural networks. We demonstrated the generality of the proposed method through its integrations with logistic regression, multinomial logistic regression and convolutional neural networks. Extensive empirical evaluations demonstrate that the proposed regularised label noise model can significantly improve, in terms of both the quality of noise parameters estimation and the classification accuracy, upon the existing ones when data is scarce, and is no worse than the existing approaches in the abundance of training data.

Appendix

In this section we show how to derive the multiplicative fixed point update equations for label noise parameters. For $\omega _{00}$ and $\omega _{01}$, we first construct a Lagrangian of Eq. (14) imposing a constraint that $\omega _{00} + \omega _{01} = 1$,

$$\begin{aligned} \mathcal {R}(\mathbf{w} , \Omega ) = \sum _{n=1}^N (1-\tilde{y}_n)\log \tilde{P}_n^0 + \tilde{y}_n\log \tilde{P}_n^1 - \sum _{j=0}^1 \sum _{k=0}^1 \alpha _{jk} \omega _{jk} + \lambda (1-\sum _{l=0}^1 \omega _{0l}) \end{aligned}$$

(25)

Taking the derivative of the above w.r.t. $\omega _{00}$ we can arrive at,

$$\begin{aligned} \lambda&= \sum _{n=1}^N \frac{(1-\tilde{y}_n)}{\tilde{P}_n^0}P_n^0 - \alpha _{00} \end{aligned}$$

(26)

Multiplying the above by $\omega _{00}$ yields,

$$\begin{aligned} \omega _{00} \lambda&= \omega _{00} \left( \sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^0} - \alpha _{00} \right) \end{aligned}$$

(27)

We can derive a similar expression for $\omega _{01}$ which turns out to be,

$$\begin{aligned} \omega _{01} \lambda&= \omega _{01} \left( \sum _{n=1}^N \tilde{y}_n\frac{P_n^0}{\tilde{P}_n^1} - \alpha _{01} \right) \end{aligned}$$

(28)

Summing Eqs. (27) and (28) and using the fact that $\omega _{00} + \omega _{01} = 1$ we can work out the expression for the Lagrange multiplier.

$$\begin{aligned} \lambda&= \omega _{00} \left( \sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^0} - \alpha _{00} \right) + \omega _{01} \left( \sum _{n=1}^N \tilde{y}_n\frac{P_n^0}{\tilde{P}_n^1} - \alpha _{01} \right) \end{aligned}$$

(29)

Substitute $\lambda$ in Eq. (29) back into Eqs. (27) and (28) gives us the multiplicative update equations for $\omega _{00}$ and $\omega _{01}$

$$\begin{aligned} \omega _{00}&= \frac{\omega _{00} \left( \sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^0} - \alpha _{00} \right) }{\omega _{00} \left( \sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^0} - \alpha _{00} \right) + \omega _{01} \left( \sum _{n=1}^N \tilde{y}_n\frac{P_n^0}{\tilde{P}_n^1} - \alpha _{01} \right) } \end{aligned}$$

(30)

$$\begin{aligned} \omega _{01}&= \frac{\omega _{01} \left( \sum _{n=1}^N \tilde{y}_n\frac{P_n^0}{\tilde{P}_n^1} - \alpha _{01} \right) }{\omega _{00} \left( \sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^0} - \alpha _{00} \right) + \omega _{01} \left( \sum _{n=1}^N (\tilde{y}_n)\frac{P_n^0}{\tilde{P}_n^1} - \alpha _{01} \right) } \end{aligned}$$

(31)

The update equations for $\omega _{10}$ and $\omega _{11}$, which can be derived similarly, turned out to be.

$$\begin{aligned} \omega _{10}&= \frac{\omega _{10} \Big (\sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^1}{\tilde{P}_n^0} - \alpha _{10} \Big )}{\omega _{10} \Big (\sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^1}{\tilde{P}_n^0} - \alpha _{10} \Big ) + \omega _{11} \Big (\sum _{n=1}^N \tilde{y}_n\frac{P_n^1}{\tilde{P}_n^1} - \alpha _{11} \Big )} \end{aligned}$$

(32)

$$\begin{aligned} \omega _{11}&= \frac{\omega _{11} \Big (\sum _{n=1}^N \tilde{y}_n \frac{P_n^1}{\tilde{P}_n^1} - \alpha _{11} \Big )}{\omega _{10} \Big (\sum _{n=1}^N (1-\tilde{y}_n)\frac{P_n^1}{\tilde{P}_n^0} - \alpha _{10} \Big ) + \omega _{11} \Big (\sum _{n=1}^N \tilde{y}_n\frac{P_n^1}{\tilde{P}_n^1} - \alpha _{11} \Big )} \end{aligned}$$

(33)