Margin maximization in spherical separation

Abstract

We face the problem of strictly separating two sets of points by means of a sphere, considering the two cases where the center of the sphere is fixed or free, respectively. In particular, for the former we present a fast and simple solution algorithm, whereas for the latter one we use the DC-Algorithm based on a DC decomposition of the error function. Numerical results for both the cases are presented on several classical binary datasets drawn from the literature.

Appendix

In this section we report the proofs of Theorems 2.1, 2.2 and 2.3.

Theorem 2.1

Let S⊆ℝⁿ be a nonempty compact set. If x ₀∈S and C≥1/(2k), then there exists an optimal solution to problem (5).

Proof

If x ₀∈S, then

$$ \|b_l-x_0\|\leq \max_{x\in S} \|b_l-x\|\stackrel {\triangle }{=}D_l\geq0, \quad l=1, \ldots, k. $$

(18)

Taking into account the definition of h, the constraint z≥q and inequality (18), we obtain

(19)

where $D\stackrel {\triangle }{=}\sum_{l=1}^{k} D_{l}$.

From the above inequality, if $C\geq\frac{1}{2k}$, the thesis follows by taking into account that h is coercive on the set $\varOmega \stackrel {\triangle }{=}\{ (x_{0},z,q)~|~x_{0}\in S \mbox{ and } 0\leq q\leq z \}$. □

Theorem 2.2

There exists an optimal solution for problem (6) with z>0.

Proof

Let $(x_{0}^{*},z^{*},q^{*},\xi^{*},\mu^{*})$ be any optimal solution for problem (6) with z ^∗=0. As a consequence q ^∗=0, $\xi^{*}_{i}=\|a_{i}-x_{0}^{*}\|^{2}$ for i=1,…,m and $\mu_{l}^{*}=0$ for l=1,…,k.

We define the sets

$$I\stackrel {\triangle }{=}\bigl\{i\mid 1\leq i\leq m,\ a_i\neq x_0^* \bigr \} \quad \mbox{and}\quad L\stackrel {\triangle }{=}\bigl\{l\mid 1\leq l\leq k,\ b_l\neq x_0^* \bigr\}, $$

which cannot be simultaneously empty, by the assumption that the sets $\mathcal {A}$ and $\mathcal {B}$ are disjoint.

Now we consider the solution

$$ \bigl(\bar{x}_0 =x_0^*, \bar{z}, \bar{q}=q^*, \bar{\xi}, \bar{\mu}\bigr), $$

(20)

with

$$\bar{z}= \min_{\scriptsize i\in I,\ l\in L} \bigl\{\|a_i-x_0^* \|^2,\|b_l-x_0^*\|^2 \bigr\}>0, $$

and the components of the vectors $\bar{\xi}$ and $\bar{\mu}$ defined as follows:

$$\mbox{(i)} \begin{cases} \bar{\xi}_i=\xi_i^*-\bar{z} &i\in I \\ \bar{\mu}_l=\mu_l^*=0& l\in L, \end{cases} $$

$$\mbox{(ii)} \begin{cases} \bar{\xi}_i=\xi_i^*-\bar{z} &i\in I \\ \bar{\xi}_j=\xi_j^*=0& \\ \bar{\mu}_l=\mu_l^*=0& l\in L \end{cases} $$

or

$$\mbox{(iii)} \begin{cases}\bar{\xi}_i=\xi_i^*-\bar{z} &i\in I \\ \bar{\mu}_l=\mu_l^*=0& l\in L \\ \bar{\mu}_j = \bar{z}, \end{cases} $$

according to the three possible cases:

• 1. (i) $x_{0}^{*}\neq a_{i}$ for i=1,…,m and $x_{0}^{*}\neq b_{l}$ for l=1,…,k;

• 2. (ii) $x_{0}^{*}= a_{j}$ for some j∈{1,…,m};

• 3. (iii) $x_{0}^{*}= b_{j}$ for some j∈{1,…,k}.

It is easy to show that the above solution (20), characterized by $\bar{z}>0$, is feasible for problem (6), with an objective function value which is not worse than that one corresponding to solution $(x_{0}^{*},z^{*},q^{*},\xi^{*},\mu^{*})$. □

Theorem 2.3

Let $(x_{0}^{*}, z^{*}, q^{*}, \xi^{*}, \mu^{*})$ be any optimal solution of problem (6), with C>1. Then the sphere $S(x_{0}^{*},R^{*})$, with $R^{*}=\sqrt{z^{*}}$, strictly separates the sets $\mathcal {A}$ and $\mathcal {B}$ if and only if q ^∗>0.

Proof

(⇒) Assume $S(x_{0}^{*},R^{*})$, with $R^{*}=\sqrt{z^{*}}$, strictly separates the sets $\mathcal {A}$ and $\mathcal {B}$, and let q ^∗=0. Thus we have

$$ \begin{cases} -z^*+\|a_i-x_0^* \|^2<0& \forall i=1,\ldots,m \\ z^*-\|b_l-x_0^*\|^2<0& \forall l=1, \ldots,k \end{cases} $$

(21)

and, consequently,

$$0<\epsilon \stackrel {\triangle }{=}\min_{1\leq i\leq m,~1\leq l\leq k } \bigl\{ z^*- \bigl\| a_i-x_0^* \bigr\|^2,\bigl\|b_l-x_0^*\bigr\|^2-z^* \bigr \} \leq z^*. $$

Note that the optimality of $(x_{0}^{*}, z^{*}, q^{*}, \xi^{*}, \mu^{*})$, together with (21), implies ξ ^∗=0 and μ ^∗=0 and the optimal objective function value equal to zero.

Define now a feasible solution $(\bar{x}_{0}, \bar{z}, \bar{q}, \bar{\xi}, \bar{\mu})$, to problem (6) as follows:

$$\bar{x}_0=x_0^*,\qquad \bar{z}=z^*,\qquad \bar{q}=\epsilon,\qquad \bar{\xi}=\xi^*=0, \qquad \bar{\mu}=\mu^*=0. $$

It is characterized by an objective function value equal to −ϵ<0, which contradicts optimality of $(x_{0}^{*}, z^{*}, q^{*}, \xi^{*}, \mu^{*})$.

(⇐) Now suppose q ^∗>0, and assume by contradiction that the sphere $S(x_{0}^{*},R^{*})$, with $R^{*}=\sqrt{z^{*}}$, does not strictly separate $\mathcal {A}$ and $\mathcal {B}$. Then at least one of the two possible cases occur:

• 1. there exists $a_{j}\in \mathcal {A}$ such that $\|a_{j}-x_{0}^{*}\|^{2}\geq z^{*}$, which implies

  $$ \xi_j^*\geq q^*-z^* + \bigl\|a_j-x_0^* \bigr\|^2 \geq q^*>0; $$

  (22)

• 2. there exists $b_{j}\in \mathcal {B}$ such that $\|b_{j}-x_{0}^{*}\|^{2}\leq z^{*}$, which implies

  $$ \mu_j^*\geq q^*+z^* - \bigl\|b_j-x_0^* \bigr\|^2 \geq q^*>0. $$

  (23)

Case 1. The value of the objective function, in correspondence to the optimal solution is

$$C \Biggl(\sum_{i=1,i\neq j}^m \xi_i^*+\xi_j^*+\sum_{l=1}^k \mu^*_l \Biggr) - q^*. $$

Now we consider the solution $(\bar{x}_{0}, \bar{z}, \bar{q}, \bar{\xi}, \bar{\mu})$ defined as follows:

$$ \begin{cases} \bar{x}_0 = x_0^* \\ \bar{z} = z^* \\ \bar{q}=0 \\ \bar{\xi}_i=\xi_i^*\quad i=1,\ldots,m, \ i\neq j \\ \bar{\xi}_j= \xi_j^*-q^* \\ \bar{\mu}= \mu^*, \end{cases} $$

(24)

which can be easily proved to be feasible to problem (6), taking into account (22). Moreover, the corresponding objective function value is

$$C \Biggl(\sum_{i=1,i\neq j}^m \xi_i^*+\xi_j^*-q^*+\sum _{l=1}^k \mu^*_l \Biggr)=C \Biggl( \sum_{i=1,i\neq j}^m \xi_i^*+ \xi_j^*+\sum_{l=1}^k \mu^*_l \Biggr) - Cq^*. $$

If C>1 then

$$C \Biggl(\sum_{i=1,i\neq j}^m \xi_i^*+\xi_j^*+\sum_{l=1}^k \mu^*_l \Biggr) - Cq^*<C \Biggl(\sum _{i=1,i\neq j}^m \xi_i^*+\xi_j^*+ \sum_{l=1}^k \mu^*_l \Biggr) - q^*, $$

which contradicts the optimality of $(x_{0}^{*}, z^{*}, q^{*}, \xi^{*}, \mu^{*})$.

Case 2. Analogous considerations hold. □