A Natural Threshold Model for Ordinal Regression

Abstract

The threshold model is one of the most commonly used ordinal regression methods. It projects patterns onto a real axis and uses a list of thresholds to divide the real axis into consecutive intervals, one for each category. We propose using a fixed sequence of natural thresholds (i.e., $0.5, 1.5, \ldots$) to simplify the model. We prove that using fixed thresholds does not degrade the performance after minor changes in the neural network structure. The natural threshold simplifies the logical complexity and implementation complexity of the model, so that the loss function can be easily customized for any special metric. We have designed loss functions for various ordinal regression evaluation metrics and achieved the best level of results on many ordinal regression tasks.

Appendix

1.1 Proof of Lemma 1

Lemma 1

Let the prediction function $Pred _{\mathcal {L}}$ be order–preserving and surjective. Then, for $\forall \alpha \in \Gamma$, $\exists -\infty< b_1< b_2< \cdots< b_{Q-1} < +\infty$, such that

$$\begin{aligned} Pred _{\mathcal {L}}(s, \alpha ) = c_q,\ s \in [b_{q-1}, b_q), \end{aligned}$$

where $q \in \{1, \ldots , Q\}, b_0 = -\infty , b_{Q} = +\infty$. In particular, we call ${\mathbf {b}} = (b_1, \ldots , b_{Q-1})$ the threshold vector of $Pred _{\mathcal {L}}$.

Proof

For $\forall \alpha \in \Gamma$, let $b_q=\sup \{ s | Pred _{\mathcal {L}}(s, \alpha ) \preceq c_q\}$, $q \in \{1, 2, \ldots , Q-1\}$. Because $c_{q-1} \preceq c_{q}$, it follows that $\{ s | Pred _{\mathcal {L}}(s, \alpha ) \preceq c_{q-1}\} \subseteq \{s | Pred _{\mathcal {L}}(s, \alpha ) \preceq c_{q}\}$ and $\sup \{ s | Pred _{\mathcal {L}}(s, \alpha ) \preceq c_{q-1}\} \le \sup \{s | Pred _{\mathcal {L}}(s, \alpha ) \preceq c_{q}\}$. That means $b_{q-1} \le b_{q}$, so $b_1 \le b_2 \le \cdots \le b_{Q-1}$.

Since $Pred _{\mathcal {L}}$ is order–preserving, we have $Pred _{\mathcal {L}}(s, \alpha ) \preceq c_q$ when $b_{q-1}< s < b_q$. By the definition of $b_{q-1}$, $Pred _{\mathcal {L}}(s, \alpha ) \preceq c_{q-1}$ is not established at this time. Thus, $Pred _{\mathcal {L}}(s, \alpha ) = c_q$ when $b_{q-1}< s < b_q$.

Since $Pred _{\mathcal {L}}$ is surjective, for $\forall c_q \in {\mathcal {Y}}$, the Lebesgue measure of $\{s | Pred _{\mathcal {L}}(s, \alpha ) = c_q\}$ is greater than 0. Thus, $-\infty< b_1< b_2< \cdots< b_{Q-1} < +\infty$.

To prove that ${\mathcal {L}}(b_{q-1}, \alpha , c_{q-1})={\mathcal {L}}(b_{q-1}, \alpha , c_q)$, we first assume that ${\mathcal {L}}(b_{q-1}, \alpha , c_{q-1}) < {\mathcal {L}}(b_{q-1}, \alpha , c_q)$. Since ${\mathcal {L}}(s, \alpha , y)$ is continuous in s, there exists $\epsilon > 0$ small enough such that ${\mathcal {L}}(b_{q-1}+\epsilon , \alpha , c_{q-1}) < {\mathcal {L}}(b_{q-1}+\epsilon , \alpha , c_q)$. Thus, $Pred _{\mathcal {L}}(b_{q-1}+\epsilon , \alpha ) = c_{q-1}$ and contradicts $Pred _{\mathcal {L}}(s, \alpha ) = c_q$ when $b_{q-1}< s < b_q$. The same contradiction can be found for ${\mathcal {L}}(b_{q-1}, \alpha , c_{q-1}) > {\mathcal {L}}(b_{q-1}, \alpha , c_q)$. So ${\mathcal {L}}(b_{q-1}, \alpha , c_{q-1})={\mathcal {L}}(b_{q-1}, \alpha , c_q)$ and $Pred _{\mathcal {L}}(b_{q-1}, \alpha ) = c_q$. $\square$

1.2 Proof of Lemma 2

Lemma 2

If a single hidden layer neural network has one input unit, Q ReLU activation hidden units, and one output unit, it can represent any strictly monotone and increasing Q–segment linear function.

Proof

For any strictly monotone and increasing Q–segment linear function f(x), let its $Q-1$ segmentation points be $t_2< t_3< \cdots < t_{Q}$ and the slopes of Q segments be $w_1,\ w_2,\ \ldots ,\ w_Q$, respectively. Then, the single hidden layer neural network

$$\begin{aligned} g(x)=&-w_1 \cdot ReLU(-x+t_2)+w_2 \cdot ReLU(x-t_2) \nonumber \\&+\sum _{i=3}^{Q} (w_i-w_{i-1}) \cdot ReLU(x-t_i) + f(t_2) \end{aligned}$$

satisfies $g(t_i)=f(t_i)$, $i \in \{2, \ldots , Q\}$. Thus, f(x) and g(x) are identical. $\square$

1.3 Proof of Theorem 1

Theorem 1

Let $({\mathcal {S}}_1, {\mathcal {L}}_1)$ be any threshold model where the prediction function $Pred _{{\mathcal {L}}_1}$ derived from ${\mathcal {L}}_1$ is order–preserving and surjective. For any given threshold vector ${\mathbf {d}}$, there exists a threshold model $({\mathcal {S}}_2, {\mathcal {L}}_2)$, which takes ${\mathbf {d}}$ as the threshold vector, and its performance is not weaker than $({\mathcal {S}}_1, {\mathcal {L}}_1)$.

Proof

According to Lemma 1, for $\forall \alpha \in \Gamma$ of the ${\mathcal {L}}_1$, there exists a threshold vector ${\mathbf {b}}$ of the $Pred _{{\mathcal {L}}_1}$ such that $Pred _{{\mathcal {L}}_1}(s, \alpha ) = c_q$ when $s \in [b_{q-1}, b_q)$. Let $\phi (x, \alpha )$ be a strictly monotone and increasing Q–segment linear function such that $\phi (b_q, \alpha )=d_q$, and the slopes of the leftmost and rightmost segments are 1. $\phi (x, \alpha )$ is determined by $\alpha$, because the segmentation points ${\mathbf {b}}$ are determined by $\alpha$, and the values at the segmentation points are ${\mathbf {d}}$.

Let

$$\begin{aligned} {\mathcal {L}}_2(s, \alpha , y) = {\mathcal {L}}_1(\phi ^{-1}(s, \alpha ), \alpha , y). \end{aligned}$$

(1)

Then, the prediction function $Pred _{{\mathcal {L}}_2}$ takes ${\mathbf {d}}$ as the threshold vector. The examples of this loss function transform are seen in Fig. 10.

Let

$$\begin{aligned} {\mathcal {S}}_2({\mathbf {x}},(\theta ,{\mathbf {w}}))&= \varphi ({\mathcal {S}}_1({\mathbf {x}},\theta ),{\mathbf {w}}), \nonumber \\ \widetilde{{\mathcal {S}}_1}({\mathbf {x}},(\theta ,{\mathbf {w}}, \alpha ))&= \phi ^{-1}(\varphi ({\mathcal {S}}_1({\mathbf {x}},\theta ),{\mathbf {w}}),\alpha ), \end{aligned}$$

(2)

where $\varphi (x,{\mathbf {w}})$ is a single hidden layer neural network with one input unit, Q ReLU activation hidden units and one output unit, and its parameter vector is ${\mathbf {w}}$. According to Eqs. 1 and 2,

$$\begin{aligned}&{\mathcal {L}}_1(\widetilde{{\mathcal {S}}_1}({\mathbf {x}},(\theta , {\mathbf {w}}, \alpha )),\alpha ,y) \nonumber \\&={\mathcal {L}}_1(\phi ^{-1}(\varphi ({\mathcal {S}}_1({\mathbf {x}},\theta ),{\mathbf {w}}),\alpha ),\alpha ,y) \nonumber \\&={\mathcal {L}}_2({\mathcal {S}}_2({\mathbf {x}},(\theta ,{\mathbf {w}})),\alpha ,y) \end{aligned}$$

(3)

is always established.

Denote

$$\begin{aligned} (\theta _1^*, \alpha _1^*)=\arg \min _{\theta , \alpha } \sum _{i=1}^{n}{\mathcal {L}}_1({\mathcal {S}}_1({\mathbf {x}}_i,\theta ),\alpha ,y_i) \end{aligned}$$

and

$$\begin{aligned} (\theta _2^*, {\mathbf {w}}_2^*, \alpha _2^*)=\arg \min _{\theta , {\mathbf {w}}, \alpha } \sum _{i=1}^{n}{\mathcal {L}}_2({\mathcal {S}}_2({\mathbf {x}}_i,(\theta , {\mathbf {w}})),\alpha ,y_i). \end{aligned}$$

According to Eq. 3, $(\theta _2^*, \, {\mathbf {w}}_2^*, \, \alpha _2^*)$ is also the optimal solution for

$$\begin{aligned} \min _{\theta ,\alpha ,{\mathbf {w}}} \sum _{i=1}^{n}{\mathcal {L}}_1(\widetilde{{\mathcal {S}}_1}({\mathbf {x}}_i,(\theta ,{\mathbf {w}},\alpha )),\alpha ,y_i). \end{aligned}$$

Therefore,

$$\begin{aligned}&\sum _{i=1}^{n}{\mathcal {L}}_1(\widetilde{{\mathcal {S}}_1}({\mathbf {x}}_i,(\theta _2^*,{\mathbf {w}}_2^*, \alpha _2^*)),\alpha _2^*,y_i) \nonumber \\&= \min _{\theta ,\alpha ,{\mathbf {w}}} \sum _{i=1}^{n} {\mathcal {L}}_1(\phi ^{-1}(\varphi ({\mathcal {S}}_1({\mathbf {x}},\theta ),{\mathbf {w}}),\alpha ),\alpha ,y) \nonumber \\&\le \min _{\theta ,\alpha } \sum _{i=1}^{n}{\mathcal {L}}_1({\mathcal {S}}_1({\mathbf {x}}_i,\theta ),\alpha ,y_i) \nonumber \\&= \sum _{i=1}^{n}{\mathcal {L}}_1({\mathcal {S}}_1({\mathbf {x}}_i,\theta _1^*),\alpha _1^*,y_i). \end{aligned}$$

The inequality sign is established because there exists a ${\mathbf {w}}_0$ such that $\varphi (x,{\mathbf {w}}_0)=\phi (x)$, according to Lemma 2. Since the threshold model $(\widetilde{{\mathcal {S}}_1}, {\mathcal {L}}_1)$ has a minimum loss value on the training set that does not exceed the minimum loss value of $({\mathcal {S}}_1, {\mathcal {L}}_1)$, it can be considered that the performance of $(\widetilde{{\mathcal {S}}_1}, {\mathcal {L}}_1)$ is not weaker than $({\mathcal {S}}_1, {\mathcal {L}}_1)$.

Fig. 10

Examples of the loss function transform. The threshold before transformation is $b=(-1,1.5,10)$, and the threshold after transformation is $d=(0.5,1.5,2.5)$. After transforming the score (that is, the x–axis), the corresponding loss functions on each threshold segment in the left picture are transformed into the same color parts in the right picture

Note that

$$\begin{aligned}&Pred _{{\mathcal {L}}_1}(\widetilde{{\mathcal {S}}_1}({\mathbf {x}},(\theta ,{\mathbf {w}}, \alpha )),\alpha ) \nonumber \\&= \arg \min _{c_q \in {\mathcal {Y}}} {\mathcal {L}}_1(\widetilde{{\mathcal {S}}_1}({\mathbf {x}},(\theta , {\mathbf {w}}, \alpha )),\alpha ,c_q) \nonumber \\&= \arg \min _{c_q \in {\mathcal {Y}}} {\mathcal {L}}_2({\mathcal {S}}_2({\mathbf {x}},(\theta ,{\mathbf {w}})),\alpha ,c_q) \nonumber \\&= Pred _{{\mathcal {L}}_2}({\mathcal {S}}_2({\mathbf {x}},(\theta ,{\mathbf {w}})),\alpha ). \end{aligned}$$

The predictions of the threshold model $({\mathcal {S}}_2, {\mathcal {L}}_2)$ and $(\widetilde{{\mathcal {S}}_1}, {\mathcal {L}}_1)$ for $\forall {\mathbf {x}} \in {\mathcal {X}}$ are the same when they have the same parameters. Then, the performances of $({\mathcal {S}}_2, {\mathcal {L}}_2)$ and $(\widetilde{{\mathcal {S}}_1}, {\mathcal {L}}_1)$ are the same. Therefore, the performance of $({\mathcal {S}}_2, {\mathcal {L}}_2)$ is not weaker than $({\mathcal {S}}_1, {\mathcal {L}}_1)$. $\square$