Sample hardness guided softmax loss for face recognition

Abstract

Face recognition (FR) has received remarkable attention for improving feature discrimination with the development of deep convolutional neural networks (CNNs). Although the existing methods have achieved great success in designing margin-based loss functions by using hard sample mining strategy, they still suffer from two issues: 1) the neglect of some training status and feature position information and 2) inaccurate weight assignment for hard samples due to the coarse hardness description. To solve these issues, we develop a novel loss function, namely Hardness Loss, to adaptively assign weights for the misclassified (hard) samples guided by their corresponding hardness, which accounts for multiple training status and feature position information. Specifically, we propose an estimator to provide the real-time training status to precisely compute the hardness for weight assignment. To the best of our knowledge, this is the first attempt to design a loss function by using multiple pieces of information about the training status and feature positions. Extensive experiments on popular face benchmarks demonstrate that the proposed method is superior to the state-of-the-art (SOTA) losses under various FR scenarios.

Appendix A: Gradient formula derivation

Let’s rewrite the formulation of the Hardness loss:

$$ \begin{array}{@{}rcl@{}} &\mathcal{L}=-\log \frac{e^{s T\left( \cos \theta_{y_{i}}\right)}}{e^{s T\left( \cos \theta_{y_{i}}\right)}+{\sum}_{j=1, j \neq y_{i}}^{n} e^{s N\left( t, \cos \theta_{j}\right)}} \\ &T\left( \cos \theta_{y_{i}}\right)=\cos \left( \theta_{y_{i}}+m\right) \\ &N\left( t, \cos \theta_{j}\right)= \left\{\begin{array}{cc} \cos \theta_{j} & \text {easy} \\ \cos \theta_{j}\left( \text{dif}_{n e g}+t_{p}^{(k)} \cdot \text{dif}_{b}+t_{p}^{(k)}\right) & \text {hard} \end{array}\right. \end{array} $$

Before we calculate the gradients w.r.t. x_i and W_j, the logits can be summarized in following three cases:

$$ \begin{array}{@{}rcl@{}} &\mathcal{L}=-\log \frac{e^{f_{y_{i}}}}{e^{f_{y_{i}}}+{\sum}_{j=1, j \neq y_{i}}^{n} e^{f_{j}}} \\ &f_{j}=\left\{\begin{array}{cc} s \cdot \cos \left( \theta_{y_{i}}+m\right) & j=y_{i} \\ s \cdot \cos \theta_{j} & j \neq y_{i}, \text { easy } \\ s \cos \theta_{j}\left( \text{dif}_{n e g}+t_{p}^{(k)} \cdot \text{dif}_{b}+t_{p}^{(k)}\right) & j \neq y_{i}, \text { hard } \end{array}\right. \end{array} $$

By chain rule, we can get:

$$ \begin{array}{@{}rcl@{}} \frac{\partial \mathcal{L}}{\partial x_{i}}=\frac{\partial \mathcal{L}}{\partial f_{j}} \cdot \frac{\partial f_{j}}{\partial \cos \theta_{j}} \cdot \frac{\partial \cos \theta_{j}}{\partial x_{i}} \\ \frac{\partial \mathcal{L}}{\partial W_{j}}=\frac{\partial \mathcal{L}}{\partial f_{j}} \cdot \frac{\partial f_{j}}{\partial \cos \theta_{j}} \cdot \frac{\partial \cos \theta_{j}}{\partial W_{j}} \end{array} $$

For $\frac {\partial {\mathscr{L}}}{\partial f_{j}}$, it is easily calculated with Softmax function:

$$ \begin{array}{@{}rcl@{}} \frac{\partial L}{\partial f_{j}}=\left\{\begin{array}{ll} a-1 & j=y_{i} \\ 1-a & j \neq y_{i} \end{array}, a=\frac{e^{f_{y_{i}}}}{e^{f_{y_{i}}}+{\sum}_{j=1, j \neq y_{i}}^{n} e^{f_{j}}}\right. \end{array} $$

For $\frac {\partial {\cos \limits } \theta _{j}}{\partial x_{i}}$ and $\frac {\partial {\cos \limits } \theta _{j}}{\partial W_{j}}$, considering ${\cos \limits } \theta _{j}=\frac {{W_{j}^{T}} x_{i}}{\left \|{W_{j}^{T}}\right \|\left \|x_{i}\right \|}={W_{j}^{T}} x_{i}$, thus we have:

$$ \begin{array}{@{}rcl@{}} \frac{\partial \cos \theta_{j}}{\partial x_{i}}=W_{j} \\ \frac{\partial \cos \theta_{j}}{\partial W_{j}}=x_{i} \end{array} $$

For $\frac {\partial f_{j}}{\partial {\cos \limits } \theta _{j}}$, it is discussed in following three cases:

when j = y_i:

$$ \begin{array}{@{}rcl@{}} \frac{\partial f_{y_{i}}}{\partial \cos \theta_{y_{i}}} &=&\frac{\partial s \cdot \cos \left( \theta_{y_{i}}+m\right)}{\partial \cos \theta_{y_{i}}} \\ &=&s \frac{\partial\left( \cos \theta_{y_{i}} \cos m-\sin \theta_{y_{i}} \sin m\right)}{\partial \cos \theta_{y_{i}}} \\ &=&s\left( \cos m+\sin m \cdot \frac{\cos \theta_{y_{i}}}{\sin \theta_{y_{i}}}\right) \\ &=&s \frac{\cos m \sin \theta_{y_{i}}+\sin m \cos \theta_{y_{i}}}{\sin \theta_{y_{i}}} \\ &=&s \frac{\sin \left( \theta_{y_{i}}+m\right)}{\sin \theta_{y_{i}}} \end{array} $$

when j≠y_i, for easy sample:

$$ \begin{array}{@{}rcl@{}} \frac{\partial f_{j}}{\partial \cos \theta_{j}}=\frac{\partial s \cdot \cos \theta_{j}}{\partial \cos \theta_{j}}=s \end{array} $$

when j≠y_i, for hard sample:

$$ \begin{array}{@{}rcl@{}} \frac{\partial f_{j}}{\partial \cos \theta_{j}} &=\frac{\partial s \cdot \cos \theta_{j}\left( \text{dif}_{n e g}+t_{p} \cdot \text{dif}_{b}+t_{p}\right)}{\partial \cos \theta_{j}} \\ &=\frac{\partial s \cdot \cos \theta_{j}\left( \cos \theta_{j}+t_{p} \cdot\left( 1+\cos \theta_{j}-\cos \left( \theta_{y_{i}}+m\right)\right)\right)}{\partial \cos \theta_{j}} \\ &=s {\Delta} \end{array} $$

where ${\Delta } = 2 {\cos \limits } \theta _{j}+t_{p}\left [1+2 {\cos \limits } \theta _{j}-{\cos \limits } \left (\theta _{y_{i}}+m\right )\right ]$, and $t_{p} =t_{p}^{(k)}$.