Output Layer Multiplication for Class Imbalance Problem in Convolutional Neural Networks

Abstract

Convolutional neural networks (CNNs) have demonstrated remarkable performance in the field of computer vision. However, they are prone to suffer from the class imbalance problem, in which the number of some classes is significantly higher or lower than that of other classes. Commonly, there are two main strategies to handle the problem, including dataset-level methods via resampling and algorithmic-level methods by modifying the existing learning frameworks. However, most of these methods need extra data resampling or elaborate algorithm design. In this work we provide an effective but extremely simple approach to tackle the imbalance problem in CNNs with cross-entropy loss. Specifically, we multiply a coefficient $\alpha > 1$ to output of the last layer in a CNN model. With this modification, the final loss function can dynamically adjust the contributions of examples from different classes during the imbalanced training procedure. Because of its simplicity, the proposed method can be easily applied in the off-the-shelf models with little change. To prove the effectiveness on imbalance problem, we design three experiments on classification tasks of increasing complexity. The experimental results show that our approach could improve the convergence rate in the training stage and/or increase accuracy for test.

Appendix

Proof for that Eq. (10) is negative with $\alpha > 1$.

Assume that there is a function

$$ f\left( x \right) = \frac{log\left( x \right)}{1 - x} , x \in \left( {0 , 1} \right). $$

(15)

By computing its derivation with $x$, we get

$$ \frac{\partial f\left( x \right)}{\partial x} = \frac{{\frac{1}{x} - 1 + log\left( x \right)}}{{\left( {1 - x} \right)^{2} }}. $$

(16)

Let $g\left( x \right)$ denote the numerator of Eq. (16), we have

$$ g\left( x \right) = \frac{1}{x} - 1 + \log \left( x \right) , x \in \left( {0 , 1} \right). $$

(17)

By computing its derivation with $x$, we have

$$ \frac{\partial g\left( x \right)}{\partial x} = \frac{1}{x}\left( {1 - \frac{1}{x}} \right). $$

(18)

Since Eq. (18) is always negative with $x \in \left( {0 , 1} \right)$, we can conclude that $g\left( x \right)$ is a decreasing function. The minimum value of $g\left( x \right)$ approaches zero, as $g\left( 1 \right) = 0$. Thus, $g\left( x \right)$ is always positive with $x \in \left( {0 , 1} \right)$. It indicates that $f\left( x \right)$ is an increasing function with $x \in \left( {0 , 1} \right)$.

Let $p_{1} = e^{{o_{k} }} /\mathop \sum_{j = 1}^{C} e^{{o_{j} }} ,$ $p_{\alpha } = e^{{\alpha o_{k} }} /\mathop \sum _{j = 1}^{C} e^{{\alpha o_{j} }}$ for short. There are $p_{1} , p_{\alpha } \in \left( {0 , 1} \right)$ and $p_{1} < p_{\alpha }$ with $\alpha > 1$, which can be inferred from the proof of Theorem 2. By considering the monotonicity of $f\left( x \right)$, we have

$$ \frac{{log\left( {p_{1} } \right)}}{{1 - p_{1} }} < \frac{{log\left( {p_{a} } \right)}}{{1 - p_{a} }}, $$

(19)

which can be transformed as

$$ \left( {1 - p_{a} } \right)log\left( {p_{1} } \right) < \left( {1 - p_{1} } \right)log\left( {p_{a} } \right). $$

(20)

Because both sides of Eq. (20) are negative, we can obtain

$$ \alpha \left( {1 - p_{a} } \right)log\left( {p_{1} } \right) < {\left( {1 - p_{1} } \right)log\left( {p_{a} } \right) , \alpha } > 1. $$

(21)

So we can conclude that Eq. (10) is negative with $\alpha > 1$.