Boundary-restricted metric learning

Abstract

Metric learning aims to learn a distance metric to properly measure the similarities between pairwise examples. Most existing learning algorithms are designed to reduce intra-class distances and meanwhile enlarge inter-class distances by critically introducing a margin between intra-class and inter-class distances. However, such learning objectives may yield boundless (distance) metric space, because their enlargements on inter-class distances are usually unconstrained. In this case, excessively enlarged inter-class distances would relatively reduce the ratio of margin to the whole distance range (i.e., the margin-range-ratio), and thus being against the initial large-margin purpose for discriminating the similarities of data pairs. To address this issue, we propose a new boundary-restricted metric (BRM), which confines the metric space by a restriction function. Such a restriction function is monotonous and gradually converges to an upper bound, which suppresses excessively large distances of data pairs and concurrently maintains the reliable discriminability. After that, the learned metric can be successfully restricted in a finite region, and thereby avoiding the reduction of margin-range-ratio. Theoretically, we prove that BRM tightens the generalization error bound of the traditional learning model without sacrificing the fitting capability or destroying the topological property of the learned metric, which implies that BRM makes a good bias-variance tradeoff for the metric learning task. Extensive experiments on toy data and real-world datasets validate the superiority of our approach over the state-of-the-art metric learning methods.

Appendix A

This section provids the detailed proofs for all theorems in Sect. 3.1 and Sect. 4.

1.1 A.1 Proof for Theorem 1

We first introduce the following Lindeberg central limit theorem (CLT) as a Lemma to prove our Theorem 1.

Lemma 1

(Lindeberg CLT (Vershynin, 2018)) Suppose $\{X_{1},\ldots ,X_{h}\}$ is a sequence of independent random variables, each with finite expected value $\mu _{i}$ and variance $\sigma _{i}^{2}$. If for any given $\epsilon >0$

$$\begin{aligned} \lim _{h\rightarrow \infty }\frac{1}{S_{h}^{2}}\sum _{i=1}^{h}\nolimits {\mathbb {E}}[(X_{i}-\mu _{i})^{2}\cdot 1_{\{\vert X_{i}-\mu _{i}\vert >\epsilon S_{n}\}}]=0, \end{aligned}$$

(17)

then the distribution of the standardized sum $1/S_{h}\sum _{i=1}^{h}(X_{i}-\mu _{i})$ converges to the standard normal distribution ${\mathcal {N}}(0,1)$, where $S_{h}^{2}=\sum _{i=1}^{h}\sigma _{i}^{2}$ and $1_{\{\cdot \}}$ is the indicator function.

Based on the above conclusion on i.n.i.d. random variables $X_{1}, X_{2}, \ldots , X_{h}$, here we prove Theorem 1 by investigating the probability of that the distance value crosses a given upper-bound. We show that such a probability is mainly determined by the boundary of the metric space.

Proof

We let $X_{i} = \vert \varphi _{i}({\varvec{x}}) - \varphi _{i}(\widehat{{\varvec{x}}})\vert ^{p}$ for $i = 1,2,\ldots ,h$ and we can obtain that ${\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i}-\mu _{i}\vert>\epsilon S_{h}\}}] \le {\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert > b^{p}({\mathcal {H}}_{v}^{u}) - \mu _{i}\}}] = 0$ for sufficiently large h. Specifically, we denote $V^{2}=1/h\sum _{i=1}^{h}\sigma _{i}^{2}>0$ and let $h=\left\lceil (b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i})^{2}/(\epsilon V)^{2}\right\rceil$, and we have that

$$\begin{aligned} \epsilon S_{h}=\epsilon \sqrt{h}V\ge \epsilon V\sqrt{(b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i})^{2}/(\epsilon V)^{2}}=b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}, \end{aligned}$$

(18)

where $b^{p}({\mathcal {H}}_{v}^{u})$ is the upper-bound of $X_{i}$ so that $b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}$ is always non-negative ($\mu _{i}$ is the mean of $X_{i}$). Then, if $\vert X_{i}-\mu _{i}\vert <b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}$, we have

$$\begin{aligned} (X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i}-\mu _{i}\vert>\epsilon S_{h}\}}=0=(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\}}. \end{aligned}$$

(19)

If $b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\le \vert X_{i}-\mu _{i} \vert \le \epsilon S_{h}$, we have

$$\begin{aligned} (X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i}-\mu _{i}\vert>\epsilon S_{h}\}}=0\le (X_{i} - \mu _{i})^{2}=(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\}.} \end{aligned}$$

(20)

Finally, if $\epsilon S_{h}< \vert X_{i}-\mu _{i}\vert$, we have

$$\begin{aligned} (X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i}-\mu _{i}\vert>\epsilon S_{h}\}}=(X_{i} - \mu _{i})^{2}=(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\}}, \end{aligned}$$

(21)

and thus we have ${\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i}-\mu _{i}\vert>\epsilon S_{h}\}}] \le {\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\}}]$ for sufficiently large h. Furthermore, as $\vert X_{i}-\mu _{i}\vert$ is always small than its upper bound $b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}$, we have ${\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >b^{p}({\mathcal {H}}_{v}^{u})-\mu _{i}\}}]=0$. Therefore, we have

$$\begin{aligned} \lim _{h\rightarrow \infty } \frac{\sum _{i=1}^{h} {\mathbb {E}}[(X_{i} - \mu _{i})^{2} \cdot 1_{\{\vert X_{i} - \mu _{i}\vert >\epsilon S_{h}\}}]}{S_{h}^{2}} \le \lim _{h\rightarrow \infty } \frac{h\cdot 0}{h\sigma _{L}^{2}} = 0, \end{aligned}$$

(22)

which implies that the Lindeberg condition in Eq. (17) is satisfied. Therefore, the standardized sum $\sum _{i=1}^{h}\vert \varphi _{i}({\varvec{x}}) - \varphi _{i}(\widehat{{\varvec{x}}})\vert ^{p}/{\widetilde{\sigma }}$ converges to the standard normal distribution .⁵ Then, for any given $\epsilon _{1}>0$, there exists sufficiently large h such that

$$\begin{aligned} \text {pr} \left[ \frac{1}{{\overline{\sigma }}\sqrt{h}} \sum _{i=1}^{h}\nolimits \vert \varphi _{i}({\varvec{x}}) - \varphi _{i}(\widehat{{\varvec{x}}})\vert ^{p} - \mu _{i} \le Z\right] \in \varDelta (\phi (Z),\epsilon _{1}), \end{aligned}$$

(23)

where $Z \in {\mathbb {R}}$ and $\phi ( \cdot )$ is the cumulative distribution function of the standard norm distribution. Meanwhile, we have

$$\begin{aligned} \text {pr}[d_{\varvec{\varphi }}({\varvec{z}},\widehat{{\varvec{z}}}) < u] = \text {pr}\left[ \sum _{i=1}^{h}\nolimits \vert \varphi _{i}({\varvec{x}}) - \varphi _{i}(\widehat{{\varvec{x}}})\vert ^{p} < hu^{p}\right] , \end{aligned}$$

(24)

so for any $\epsilon _{1} > 0$, there exists sufficiently large h such that

$$\begin{aligned} \text {pr}[d_{\varvec{\varphi }}({\varvec{z}},\widehat{{\varvec{z}}}) < u] \in \varDelta (\phi (\sqrt{h}(u^{p} - {\overline{\mu }})/{\overline{\sigma }}), \epsilon _{1}). \end{aligned}$$

(25)

As $\phi (\cdot )$ is monotonically increasing and $\epsilon _{1}$ is a given sufficiently small number, $\text {sup}_{\varvec{\varphi }\in {\mathcal {H}}_{v}^{u}}\left\{ \text {pr}\left[ d_{\varvec{\varphi }}({\varvec{z}},\widehat{{\varvec{z}}}) < u\right] \right\}$ is dominated by $\sqrt{h}(u^{p} - {\overline{\mu }})/{\overline{\sigma }}$. According to the law of large numbers (Vershynin, 2018), it follows that for any $\varvec{\varphi }\in {\mathcal {H}}_{v}^{u}$ there exists a sufficiently large N making

$$\begin{aligned} |{\overline{\mu }}-m_{N}|<\epsilon _{2} \text {and} |{\overline{\sigma }}\sqrt{h}-\varSigma _{N}|<\epsilon _{2}, \end{aligned}$$

(26)

with at least probability $1-\epsilon _{2}$, where the sample mean $m_{N} = (1/N) \sum _{j=1}^{N} d_{\varvec{\varphi }}({\varvec{z}}_{j}, \widehat{{\varvec{z}}}_{j})$ and sample variance $\varSigma _{N}^{2} = ((1/N)) \sum _{j=1}^{N} (d_{\varvec{\varphi }}({\varvec{z}}_{j}, \widehat{{\varvec{z}}}_{j})-m_{N})^{2}$. Then we have that there exists sufficiently small $\epsilon _{1}$ and $\epsilon _{2}$ such that for any given $\delta \in \min (1,v^{p} - u^{p})$

$$\begin{aligned} \sup _{\varvec{\varphi }\in {\mathcal {H}}_{v}^{u}}\{\text {pr}\left[ d_{\varvec{\varphi }}({\varvec{z}},\widehat{{\varvec{z}}}) < u\right] \}\in \varDelta \left( \phi \left[ \sqrt{h}\left( u^{p} - m_{N}\right) /\varSigma _{N}\right] ,\delta \right) . \end{aligned}$$

(27)

Now we only have to consider the minimal value of the positive term $\left( m_{N} - u^{p}\right) /\varSigma _{N}$ under the constraint $d_{\varvec{\varphi }}({\varvec{z}}_{j}, \widehat{{\varvec{z}}}_{j}) > v > u$ for $j = 1,2,\ldots ,N$. To be specific, it holds that

$$\begin{aligned}&\left( m_{N}-u^{p}\right) /\varSigma _{N}\nonumber \\&\quad =\left( m_{N}-v^{p}+v^{p}-u^{p}\right) /\varSigma _{N}\nonumber \\&\quad \ge \left( m_{N} - v^{p} + v^{p} - u^{p}\right) /\min (m_{N} - v^{p}, b^{p}({\mathcal {H}}_{v}^{u}) - m_{N})\nonumber \\&\quad \ge \left( m_{N} - v^{p} + v^{p} - u^{p}\right) /(m_{N} - v^{p})\nonumber \\&\quad =(t + v^{p} - u^{p})/t\nonumber \\&\quad =(1 + (v^{p} - u^{p})/t)\nonumber \\&\quad \ge (1 + 2(v^{p} - u^{p})/(b^{p}({\mathcal {H}}_{v}^{u}) - v^{p})), \end{aligned}$$

(28)

where $t = m_{N} - v^{p}\in (0, \frac{1}{2}(b^{p}({\mathcal {H}}_{v}^{u})-v^{p})]$, and $m_{N}$ is necessarily included in $(v^{p}, \frac{1}{2}(b^{p}({\mathcal {H}}_{v}^{u})+v^{p}))$. By combining the results in Eqs. (27) and (28), we thus get

$$\begin{aligned} \sup _{\varvec{\varphi }\in {\mathcal {H}}_{v}^{u}} \left\{ \text {pr}\left[ d_{\varvec{\varphi }}({\varvec{z}},\widehat{{\varvec{z}}}) < u\right] \right\} \in \varDelta \left( \phi \left\{ \psi \left[ b({\mathcal {H}}_{v}^{u})\right] \right\} ,\delta \right) , \end{aligned}$$

(29)

where $\psi \left[ b({\mathcal {H}}_{v}^{u})\right] = [\sqrt{h}(1 + 2(v^{p} - u^{p})/(v^{p} - b^{p}({\mathcal {H}}_{v}^{u})))]$. Here $\psi \left[ b({\mathcal {H}}_{v}^{u})\right]$ is a monotonically increasing function w.r.t. the boundary $b({\mathcal {H}}_{v}^{u})$. The proof is completed. $\square$

1.2 A.2 Proof for Theorem 2

Proof

The Non-negativity and Symmetry can be trivially achieved by the definition of BRM. Here we prove that ${\mathcal {D}}_{\varvec{\varphi }}(\cdot ,\cdot )$ has the triangle property when ${\mathcal {R}}'$ is monotonically decreasing. Specifically, for any given $\varvec{\alpha },\varvec{\beta },{\varvec{\gamma }} \in {\mathbb {R}}^{d}$, we invoke the mean value theorem (Rudin, 1964) and have that

$$\begin{aligned}&{\mathcal {R}}(\vert \varphi _{i}(\varvec{\alpha })-\varphi _{i}(\varvec{\beta })\vert )+{\mathcal {R}}(\vert \varphi _{i}(\varvec{\beta })-\varphi _{i}(\varvec{\gamma })\vert )\nonumber \\&\quad ={\mathcal {R}}(Q_{i})+{\mathcal {R}}(T_{i})-{\mathcal {R}}(0)\nonumber \\&\quad ={\mathcal {R}}(\max (Q_{i},T_{i}))+\min (Q_{i},T_{i}){\mathcal {R}}'(\xi _{1})\nonumber \\&\quad \ge {\mathcal {R}}(\max (Q_{i},T_{i}))+\min (Q_{i},T_{i}){\mathcal {R}}'(\max (Q_{i},T_{i}))\nonumber \\&\quad \ge {\mathcal {R}}(\max (Q_{i},T_{i}))+\min (Q_{i},T_{i}){\mathcal {R}}'(\max (Q_{i},T_{i}))+\varTheta (\xi _{2})\nonumber \\&\quad ={\mathcal {R}}(\max (Q_{i},T_{i})+\min (Q_{i},T_{i}))\nonumber \\&\quad ={\mathcal {R}}(\vert \varphi _{i}(\varvec{\alpha })-\varphi _{i}(\varvec{\beta })\vert +\vert \varphi _{i}(\varvec{\beta })-\varphi _{i}(\varvec{\gamma })\vert )\nonumber \\&\quad \ge {\mathcal {R}}(\vert \varphi _{i}(\varvec{\alpha })-\varphi _{i}(\varvec{\gamma })\vert ), \end{aligned}$$

(30)

where the real numbers $Q_{i} = \vert \varphi _{i}(\varvec{\alpha }) - \varphi _{i}(\varvec{\beta })\vert$, $T_{i} = \vert \varphi _{i}(\varvec{\beta }) - \varphi _{i}(\varvec{\gamma })\vert$, $\xi _{1} \in [0,\min (Q_{i}, T_{i})]$, $\xi _{2} \in [0,\max (Q_{i}, T_{i})]$, and $\varTheta (\xi _{2}) = (1/2)\min (Q_{i}^{2},T_{i}^{2}){\mathcal {R}}''(\xi _{2}) \le 0$. Then we have that

$$\begin{aligned}&{\mathcal {D}}_{\varvec{\varphi }}(\varvec{\alpha }, \varvec{\beta })+{\mathcal {D}}_{\varvec{\varphi }}(\varvec{\beta }, \varvec{\gamma })\nonumber \\&\quad =\left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(Q_{i})\right] ^{p}\right) ^{ 1/p} +\left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(T_{i})\right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad \ge \left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(Q_{i})+{\mathcal {R}}(T_{i})\right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad \ge \left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(\vert \varphi _{i}(\varvec{\alpha })-\varphi _{i}(\varvec{\gamma })\vert )\right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad ={\mathcal {D}}_{\varvec{\varphi }}(\varvec{\alpha }, \varvec{\gamma }). \end{aligned}$$

(31)

Finally, for any given ${\mathcal {D}}_{\varvec{\varphi }}(\varvec{\alpha }, \varvec{\beta }) = 0$, and any given $k \in {\mathbb {N}}_{h}$, we have ${\mathcal {R}}(\vert \varphi _{k}(\varvec{\alpha })-\varphi _{k}(\varvec{\beta })\vert )=0,$ and thus it holds that $[\varphi _{1}(\varvec{\alpha }),\ldots ,\varphi _{h}(\varvec{\alpha })] = [\varphi _{1}(\varvec{\beta }),\ldots ,\varphi _{h}(\varvec{\beta })]$. By further invoking the invertibility of the mapping $\varvec{\varphi }$, we have that $\varvec{\alpha } = \varvec{\beta }$ which completes the proof. $\square$

1.3 A.3 Proof for Theorem 3

Proof

We let $\widehat{\varvec{\varphi }}(\cdot ) = c\varvec{\varphi }(\cdot )$ ($c > 0$) and employ the Taylor expansion (Rudin, 1964) on each restriction function, and we get

$$\begin{aligned}&{\mathcal {D}}_{\widehat{\varvec{\varphi }}}({\varvec{x}},\widehat{{\varvec{x}}})\nonumber \\&\quad =\left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(c\vert \varphi _{i}({\varvec{x}})-\varphi _{i}(\widehat{{\varvec{x}}})\vert )\right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad =\left( \frac{1}{h}\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}(0)+c\vert \varphi _{i}({\varvec{x}})-\varphi _{i}(\widehat{{\varvec{x}}})\vert {\mathcal {R}}'(0)+o(c)\right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad =\frac{c}{h}{\mathcal {R}}'(0)\Vert \varvec{\varphi }({\varvec{x}})-\varvec{\varphi }(\widehat{{\varvec{x}}})\Vert _{1}+o(c). \end{aligned}$$

(32)

According to the homogeneity of vector norm (Meyer, 2000), it follows that there exists $a_{1} > a_{0} > 0$ such that $\forall {\varvec{x}},\widehat{{\varvec{x}}}\in {\mathbb {R}}^{d}$

$$\begin{aligned} \frac{a_{0}}{h}\Vert \varvec{\varphi }({\varvec{x}}) - \varvec{\varphi }(\widehat{{\varvec{x}}})\Vert _{1}\le d_{\varvec{\varphi }}({\varvec{x}},\widehat{{\varvec{x}}})\le \frac{a_{1}}{h}\Vert \varvec{\varphi }({\varvec{x}}) - \varvec{\varphi }(\widehat{{\varvec{x}}})\Vert _{1}, \end{aligned}$$

(33)

so we have that

$$\begin{aligned} {\left\{ \begin{array}{ll} {\mathcal {D}}_{\widehat{\varvec{\varphi }}}({\varvec{x}}^{-},\widehat{{\varvec{x}}}^{-})\ge \frac{c}{a_{1}}{\mathcal {R}}'(0)d_{\varvec{\varphi }}({\varvec{x}}^{-},\widehat{{\varvec{x}}}^{-})+o(c),\\ {\mathcal {D}}_{\widehat{\varvec{\varphi }}}({\varvec{x}}^{+},\widehat{{\varvec{x}}}^{+})\le \frac{c}{a_{0}}{\mathcal {R}}'(0)d_{\varvec{\varphi }}({\varvec{x}}^{+},\widehat{{\varvec{x}}}^{+})+o(c). \end{array}\right. } \end{aligned}$$

(34)

Then for $u=a_{0}/{\mathcal {R}}'(0)$ and $v=a_{1}/{\mathcal {R}}'(0)$, we have

$$\begin{aligned} {\mathcal {D}}_{\widehat{\varvec{\varphi }}}({\varvec{x}}^{-},\widehat{{\varvec{x}}}^{-})\ge cv + o(c)>cu + o(c)\ge {\mathcal {D}}_{\widehat{\varvec{\varphi }}}({\varvec{x}}^{+},\widehat{{\varvec{x}}}^{+}), \end{aligned}$$

(35)

which completes the proof by letting c be sufficiently small. $\square$

1.4 A.4 Proof for Theorem 4

We first introduce the following McDiarmids inequality as a Lemma to prove our Theorem 4.

Lemma 2

(McDiarmid Inequality (Meyer, 2000)) For independent random variables $t_{1},t_{2},\ldots ,t_{n} \in {\mathcal {T}}$ and a given function $\omega :{\mathcal {T}}^{n} \rightarrow {\mathbb {R}}$, if $\forall v_{i}' \in {\mathcal {T}}$ ($i = 1,2,\ldots ,n$), the function satisfies

$$\begin{aligned} \vert \omega (t_{1},\ldots ,t_{i},\ldots ,t_{n}) - \omega (t_{1},\ldots ,t_{i}',\ldots ,t_{n})\vert \le \rho _{i}, \end{aligned}$$

(36)

then for any given $\mu > 0$, it holds that $\text {pr}\{\vert \omega (t_{1},\ldots ,t_{n}) - {\mathbb {E}}[\omega (t_{1},\ldots ,t_{n})]\vert > \mu \} \le 2\text {e}^{-2\mu ^{2}/\sum _{i=1}^{n}\rho _{i}^{2}}$.

We prove Theorem 4 by analyzing the perturbation [i.e., $\rho _{i}$ in the above Eq. (36)] of the loss function ${\mathcal {L}}$.

Proof

Firstly, we denote that

$$\begin{aligned} \omega = \frac{1}{N}\sum _{i=1}^{N}\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i}), \end{aligned}$$

(37)

and

$$\begin{aligned} \omega _{(k)} = \frac{1}{N} \left[ \sum _{i=1,i\ne k}^{N} \ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i}) + \ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{a}}_{k},\widehat{{\varvec{a}}}_{k});b_{k}) \right] , \end{aligned}$$

(38)

where $({\varvec{a}}_{k},\widehat{{\varvec{a}}}_{k})$ is an arbitrary data pair from the sample space with similarity label $b_{k}$. Then we have that

$$\begin{aligned}&|\omega -\omega _{(k)}|\nonumber \\&\quad =\frac{1}{N}|\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{k},\widehat{{\varvec{x}}}_{k});y_{k})-\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{a}}_{k},\widehat{{\varvec{a}}}_{k});b_{k})|\nonumber \\&\quad \le \frac{1}{N}\max (\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{k},\widehat{{\varvec{x}}}_{k});y_{k}),\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{a}}_{k},\widehat{{\varvec{a}}}_{k});b_{k}))\nonumber \\&\quad \le \frac{1}{N}\max (\ell ((1-c_{1})B;1),\ell ((0-c_{0})B;0)). \end{aligned}$$

(39)

Meanwhile, we have

$$\begin{aligned}&\frac{1}{N} \sum _{i=1}^{N}\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i}) - {\mathbb {E}}_{{\mathcal {X}}}\left( \frac{1}{N} \sum _{i=1}^{N}\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i})\right) \nonumber \\&\quad = \frac{1}{N} \sum _{i=1}^{N}\ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i}) - {\mathbb {E}}_{({\varvec{x}},\widehat{{\varvec{x}}})}\left[ \ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}},\widehat{{\varvec{x}}});y_{({\varvec{x}},\widehat{{\varvec{x}}})})\right] \nonumber \\&\quad ={\mathcal {L}}(\varvec{\varphi })-\widetilde{{\mathcal {L}}}(\varvec{\varphi }). \end{aligned}$$

(40)

By Lemma 2, we let that

$$\begin{aligned} \rho _{i}=\frac{1}{N}\max (\ell ((1-c_{1})B;1),\ell ((0-c_{0})B;0)), \end{aligned}$$

(41)

for all $i=1,2,\ldots ,N$, and we get

$$\begin{aligned}&\text {pr}\left\{ |{\mathcal {L}}(\varvec{\varphi })-\widetilde{{\mathcal {L}}}(\varvec{\varphi })|<\theta (B)\sqrt{[\text {ln}(2/\delta )]/(2N)}\right\} \nonumber \\&\quad =1 - 2\text {e}^{-2\mu ^{2}/\sum _{i=1}^{n}\rho _{i}^{2}}\nonumber \\&\quad \ge 1 - 2\text {e}^{\frac{-2N(\theta (B)\sqrt{[\text {ln}(2/\delta )]/(2N)})^{2}}{\max ^{2}(\ell ((1-c_{1})B;1),\ell ((0-c_{0})B;0))}}\nonumber \\&\quad =1 - 2\text {e}^{-2N\left( \sqrt{[\text {ln}(2/\delta )]/(2N)}\right) ^{2}}\nonumber \\&\quad =1 - 2\text {e}^{-\text {ln}(2/\delta )}\nonumber \\&\quad =1-\delta , \end{aligned}$$

(42)

where the real-valued monotonically increasing function $\theta (B)=\max (\ell ((1-c_{1})B;1),\ell (-c_{0}B;0))$. The proof is completed. $\square$

1.5 A.5 Proof of Lipschitz-Continuity

Here we demonstrate that our learning objective ${\mathcal {F}}(\varvec{\varphi })$ is always Lipschitz continuous based on the Lipschitz-continuity of $\varvec{\varphi }(\cdot )$, $\ell (\cdot )$, $\varOmega (\cdot )$, and ${\mathcal {R}}(\cdot )$. To be more specific, for any two given $\varvec{\varphi }$ and $\widetilde{\varvec{\varphi }}$, we have

$$\begin{aligned}&\vert {\mathcal {F}}(\varvec{\varphi })-{\mathcal {F}}(\widetilde{\varvec{\varphi }})\vert \nonumber \\&\quad =\vert 1/N\sum _{i=1}^{N}\nolimits \ell ({\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i})+\lambda \varOmega (\varvec{\varphi })\nonumber \\&\quad \quad -1/N\sum _{i=1}^{N}\nolimits \ell ({\mathcal {D}}_{\widetilde{\varvec{\varphi }}}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i});y_{i})-\lambda \varOmega (\widetilde{\varvec{\varphi }})\vert \nonumber \\&\quad \le L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits \vert {\mathcal {D}}_{\varvec{\varphi }}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i})-{\mathcal {D}}_{\widetilde{\varvec{\varphi }}}({\varvec{x}}_{i},\widehat{{\varvec{x}}}_{i})\vert +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad =L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits \vert \left( 1/h\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}\left( \vert \varphi _{i}({\varvec{x}}_{i})-\varphi _{i}(\widehat{{\varvec{x}}}_{i})\vert \right) \right] ^{p}\right) ^{ 1/p}\nonumber \\&\quad \quad -\left( 1\sum _{i=1}^{h}\nolimits \left[ {\mathcal {R}}\left( \vert {\widetilde{\varphi }}_{i}({\varvec{x}}_{i})-{\widetilde{\varphi }}_{i}(\widehat{{\varvec{x}}}_{i})\vert \right) \right] ^{p}\right) ^{ 1/p}\vert +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad =L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits \vert \omega (\vert \varvec{\varphi }({\varvec{x}}_{i})-\varvec{\varphi }(\widehat{{\varvec{x}}}_{i})\vert )-\omega (\vert \widetilde{\varvec{\varphi }}({\varvec{x}}_{i})-\widetilde{\varvec{\varphi }}(\widehat{{\varvec{x}}}_{i})\vert )\vert +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad \le L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}\Vert \vert \varvec{\varphi }({\varvec{x}}_{i})-\varvec{\varphi }(\widehat{{\varvec{x}}}_{i})\vert -\vert \widetilde{\varvec{\varphi }}({\varvec{x}}_{i})-\widetilde{\varvec{\varphi }}(\widehat{{\varvec{x}}}_{i})\vert \Vert _{2}\nonumber \\&\quad \quad +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad \le L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}\Vert \varvec{\varphi }({\varvec{x}}_{i})-\varvec{\varphi }(\widehat{{\varvec{x}}}_{i})+\widetilde{\varvec{\varphi }}(\widehat{{\varvec{x}}}_{i})-\widetilde{\varvec{\varphi }}({\varvec{x}}_{i})\Vert _{2}\nonumber \\&\quad \quad +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad \le L_{1}\frac{1}{N}\sum _{i=1}^{N}\nolimits B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}\left( \Vert \varvec{\varphi }({\varvec{x}}_{i})-\widetilde{\varvec{\varphi }}({\varvec{x}}_{i})\Vert _{2}+\Vert \varvec{\varphi }(\widehat{{\varvec{x}}}_{i})-\widetilde{\varvec{\varphi }}(\widehat{{\varvec{x}}}_{i})\Vert _{2}\right) \nonumber \\&\quad \quad +\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad \le 2L_{0}L_{1}B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}+\lambda L_{2}\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}\nonumber \\&\quad =(2L_{0}L_{1}B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}+\lambda L_{2})\Vert \varvec{\varphi }-\widetilde{\varvec{\varphi }}\Vert _{2}, \end{aligned}$$

(43)

which implies that $(2L_{0}L_{1}B^{-(1-p)^{2}}L_{{{{\mathcal {R}}}}}+\lambda L_{2})$ is a valid Lipschitz constant of our learning objective ${\mathcal {F}}$.