Equiangular Basis Vectors: A Novel Paradigm for Classification Tasks

Abstract

In this paper, we propose Equiangular Basis Vectors (EBVs) as a novel training paradigm of deep learning for image classification tasks. Differing from prominent training paradigms, e.g., k-way classification layers (mapping the learned representations to the label space) and deep metric learning (quantifying sample similarity), our method generates normalized vector embeddings as "predefined classifiers", which act as the fixed learning targets corresponding to different categories. By minimizing the spherical distance of the embedding of an input between its categorical EBV in training, the predictions can be obtained by identifying the categorical EBV with the smallest distance during inference. More importantly, by directly adding EBVs corresponding to newly added categories of equal status on the basis of existing EBVs, our method exhibits strong scalability to deal with the large increase of training categories in open-environment machine learning. In experiments, we evaluate EBVs on diverse computer vision tasks with large-scale real-world datasets, including classification on ImageNet-1K, object detection on COCO, semantic segmentation on ADE20K, etc. We further collected a dataset consisting of 100,000 categories to validate the superior performance of EBVs when handling a large number of categories. Comprehensive experiments validate both the effectiveness and scalability of our EBVs. Our method won the first place in the 2022 DIGIX Global AI Challenge, code along with all associated logs are open-source and available at https://github.com/aassxun/Equiangular-Basis-Vectors.

Appendix A

Proof of Why A Unit Hypersphere is Effective: Following Sects. 3.3 and 3.4, we have M sample-label pairs in N classes. $\varvec{W}\in \mathbb {R}^{d\times N}$ denotes the EBVs matrix while $\hat{\varvec{w}}_{i}\in \mathbb {R}^{d}$ represents each categorical EBV, where $i \in \{1,2,\ldots ,N\}$ and $\Vert \hat{\varvec{w}}_{i}\Vert =1$. Since we have already assumed that all samples are well-separated, we directly use $\hat{\varvec{w}}_{i}$ to represent the i-th class’ feature. Please allow me to emphasize once again that the categorical EBV for each class remains unchanged throughout the training process, which distinguishes it from those prototype-based methods.

As we assume that every class has the same sample number, therefore, the definition of the softmax loss is:

$$\begin{aligned} \mathcal {L}_{\texttt {softmax}} = -\frac{1}{N}\sum _{i=1}^N{\log \frac{e^{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{i}}}{\sum _{j=1}^N{e^{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}}}}\,. \end{aligned}$$

(15)

Additionally, as we adopt a temperature hyper-parameter $\tau$ within the softmax loss, then its formulation is change into:

$$\begin{aligned} \mathcal {L}_{\mathcal {S}} = -\frac{1}{N}\sum _{i=1}^N{\log \frac{e^{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{i}}{\tau }}}{\sum _{j=1}^N {e^{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}{\tau }}}}}\,. \end{aligned}$$

(16)

By dividing $e^{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{i}}{\tau }} = e^{\frac{1}{\tau }}$ from both the numerator and denominator:

$$\begin{aligned} \begin{aligned} \mathcal {L}_\mathcal {S}&= -\frac{1}{N}\sum _{i=1}^N {\log \frac{1}{1 + \sum _{j=1,j\ne i}^N {e^{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}{\tau } - \frac{1}{\tau }}}}}\\&= \frac{1}{N}\sum _{i=1}^N {\log \left( {1 + \sum _{j=1,j\ne i}^N {e^{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j} - 1 }{\tau }}}} \right) }\,. \end{aligned} \end{aligned}$$

(17)

Evidently, $f(x) = e^x$ is a convex function and $\frac{1}{N}\sum _{i=1}^N {e^{x_i}} \ge e^{\frac{1}{N}\sum _{i=1}^{N}{x_i}}$. Then we have:

$$\begin{aligned} \begin{aligned} \mathcal {L}_\mathcal {S} \ge \frac{1}{N}\sum _{i=1}^N {\log \left( {1 + (N-1) e^{\frac{1}{N-1} \sum \limits _{j=1,j\ne i}^N{( \frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j} - 1 }{\tau } )}}}\right) }. \end{aligned} \end{aligned}$$

(18)

The equality holds if and only if all $\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}{\tau }, 1\le i < j \le N$ have the same value, i.e., features from different classes have the same distance. As it has been proved that with a fixed common angle, the maximum number of equiangular lines is linearly correlated with the dimension d as $d \rightarrow \infty$ (Jiang et al., 2021). Since we have consider a large number of categories, which is much bigger than the dimension d of feature, (e.g., the class number equals 1000 while the dimension of feature equals 100 in Table 3, the class number equals 100,000 while the dimension of feature equals 5000 in Table 9. Note that ‘EBVs Dimension’ equals the dimension of feature.) this equality actually cannot hold in practice. Following (Wang et al., 2017), we then take feature dimension into consideration and improve the previous inequality.

Similar with $f(x) = e^x$, the softplus function $s(x) = \log (1 + C e^x)$ is also a convex function when $C>0$, so that $\frac{1}{N}\sum _{i=1}^N{\log (1 + C e^{x_i})} \ge \log (1 + C e^{\frac{1}{N}\sum _{i=1}^{N}{x_i}})$, then we have:

$$\begin{aligned} \begin{aligned}&\mathcal {L}_\mathcal {S} \ge \log \left( 1+ \left( N-1 \right) e^{\frac{1}{N(N-1)}\sum \limits _{i=1}^N\sum \limits _{j=1,j\ne i}^n{(\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j} - 1}{\tau })}} \right) \\&= \log \left( 1+ \left( N-1 \right) e^{\left( \frac{1}{N(N-1)}\sum \limits _{i=1}^N\sum \limits _{j=1,j\ne i}^N{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}{\tau }}\right) - \frac{1}{\tau }} \right) \,. \end{aligned} \end{aligned}$$

(19)

This equality holds if and only if $\forall \varvec{w}_i$, the sums of distances to other class’ weight $\sum _{j=1,j\ne i}^N{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}$ are all the same.

Note that

$$\begin{aligned} \left| \left| \sum _{i=1}^N{\varvec{w}_i}\right| \right| _2^2 = N + \sum _{i=1}^N\sum _{j=1,j\ne i}^N{\varvec{w}_i^{\top } \varvec{w}_j}, \end{aligned}$$

(20)

so

$$\begin{aligned} \sum _{i=1}^N\sum _{j=1,j\ne i}^N{\frac{\hat{\varvec{w}}_{i}^{\top } \hat{\varvec{w}}_{j}}{\tau }} \ge -\frac{N}{\tau }. \end{aligned}$$

(21)

The equality holds if and only if $\sum _{i=1}^N{\varvec{w}_i}=\textbf{0}$. Thus,

$$\begin{aligned} \begin{aligned} \mathcal {L}_\mathcal {S}&\ge \log \left( 1+ \left( N-1 \right) e^{-\frac{N}{\tau N(N-1)}-\frac{1}{\tau }}\right) \\&=\log \left( 1+ \left( N-1 \right) e^{-\frac{N}{\tau (N-1)}}\right) . \end{aligned} \end{aligned}$$

(22)

Taking $N=1000$ as an example, we have mentioned that $\tau$ is set as 0.07 in Sect. 4.1.1. Therefore the lower bound is around 0.00062. That is, the temperature hyper-parameter has already dealt with the problem that the softmax loss will be trapped at a very high value on training set if we normalize the features and weights to 1, and it will be fine to keep the predefined EBVs on the surface of a normalized hypersphere.