Long-tailed image recognition through balancing discriminant quality

Abstract

Long-tailed image recognition is a challenging task in real scenes with large-scale data. Popular strategies, such as loss reweighting and data resampling, aim to reduce the model bias toward head classes. Specifically, different loss reweighting approaches explore various endogenous or exogenous measures. In this paper, we study a new endogenous measure called discriminant quality (DQ) by considering validation accuracy and discriminant uncertainty. DQ takes advantage of continuous information over a period of time. It is more robust than instantaneous information because of the mitigation of measuring instability caused by random perturbations during training. Additionally, the weight of each class is automatically rebalanced based on DQ. Consequently, the class weight supports the design of a dynamic updating strategy for the significance of the DQ difference. Experiments on MNIST-LT, CIFAR-100-LT, ImageNet-LT, and Places-LT demonstrated the superiority of DQ over state-of-the-art ones in terms of prediction accuracy.

Appendix: Discussion of cross-entropy based losses

We discuss seven types of cross-entropy based losses: TWL King and Zeng (2001), FL Lin et al. (2017), CBL Cui et al. (2019), EQL Tan et al. (2020), PolyLoss Leng et al. (2022), CDB Sinha et al. (2020), and DQBL. They fall into two categories: class-wise weighted loss (CWL) and sample-wise weighted loss (SWL). CWL examines some properties of each class in the training data or validation data, and the weights are assigned class by class, whereas SWL does not ignore the classification probability (i.e., p) of an instance, and p is the basis for adjusting the sample weight during the training process. For instance, TWL assigns fixed weights to each class using the number of training samples, and aims to compensate for differences between the sample distribution and population distribution. CBL considers the effective number of instances, thereby adjusting the weights of each class. CDB measures the class difficulty by considering the overall evaluation accuracy. In DQBL, the class weights are adjusted based on DQ. FL and PolyLoss belong to the SWL family because they adjust the sample weights according to p. FL widens the gap of the relative weights between easy and hard samples, thus improving the classification accuracy of hard samples. PolyLoss considers the sample weights as the polynomial functions of p; additionally, the weights are a type of sum, that is, the Taylor expansion of CEL or FL.

Next, we examine the flexibility of adjusting the weights for each loss, which is characterized by the weight adjusting frequency during training. Regarding the two SWLs, that is, FL and PolyLoss, they both adjust the weight batch by batch. To narrow CWL down, TWL and CBL assign a fixed weight during the entire training process. These fixed weighting losses only consider the endogenous property of the training data, that is, the distribution of each class. Softmax EQL also belongs to CWL. Because of the random factor, softmax EQL modifies the class weights batch by batch. In the machine learning domain, a general hypothesis is that the distributions of training and validation data are consistent. In this paper, we satisfied the hypothesis because the training and validation data were randomly drawn from the same sample space. Although the distribution of validation data seems to be of little use, there are still some exogenous properties of these data, such as class difficulty Sinha et al. (2020) and misclassification cost Wu et al. (2019). In DQBL, we consider both validation accuracy and discriminant uncertainty to measure DQ throughout training. The class weight of DQBL is rebalanced epoch by epoch, which is similar to the reweighting mechanism of CDB. To summarize, the comparison among these losses in terms of flexibility is:

$$\begin{aligned} (\text {FL}, \text {PolyLoss}, \text {EQL})> (\text {DQBL}, \text {CDB}) > \text {(TWL, CBL)}. \end{aligned}$$