Discriminant feature extraction for image recognition using complete robust maximum margin criterion

Abstract

Maximum margin criterion (MMC) is a promising feature extraction method proposed recently to enhance the well-known linear discriminant analysis. However, due to the maximization of L2-norm-based distances between different classes, the features extracted by MMC are not robust enough in the sense that in the case of multi-class, the faraway class pair may skew the solution from the desired one, thus leading the nearby class pair to overlap. Aiming at addressing this problem to enhance the performance of MMC, in this paper, we present a novel algorithm called complete robust maximum margin criterion (CRMMC) which includes three key components. To deemphasize the impact of faraway class pair, we maximize the L1-norm-based distance between different classes. To eliminate possible correlations between features, we incorporate an orthonormality constraint into CRMMC. To fully exploit discriminant information contained in the whole feature space, we decompose CRMMC into two orthogonal complementary subspaces, from which the discriminant features are extracted. In such a way, CRMMC can iteratively extract features by solving two related constrained optimization problems. To solve the resulting mathematical models, we further develop an effective algorithm by properly combining polarity function and optimal projected gradient method. Extensive experiments on both synthesized and benchmark datasets verify the effectiveness of the proposed method.

Appendices

Appendix 1: Proof of Proposition 1

Proof

First, we prove the convergence of Algorithm 1. According to the step 3 of Algorithm 1, it can be seen that the updating for x is non-increasing. That is to say, for each t, we have

$$\begin{aligned} g( {x^{( t)}})-\sum {p_i^{( t)} a_i^\mathrm{T} } x^{( t)}\ge g( {x^{( {t+1})}})-\sum {p_i^{( t)} a_i^\mathrm{T} } x^{( {t+1})}. \end{aligned}$$

(18)

Additionally, note that $|{a_i^\mathrm{T} x^{( {t+1})}}|=sign( {a_i^\mathrm{T} x^{( {t+1})}})a_i^\mathrm{T} x^{( {t+1})}\ge sign( {a_i^\mathrm{T} x^{( t)}})a_i^\mathrm{T} x^{( {t+1})}=p_i^{( t)} a_i^\mathrm{T} x^{( {t+1})}$ and $| {a_i^\mathrm{T} x^{( t)}}|=sign( {a_i^\mathrm{T} x^{( t)}})a_i^\mathrm{T} x^{( t)}=p_i^{( t)} a_i^\mathrm{T} x^{( t)}$; it follows that

$$\begin{aligned} p_i^{( t)} a_i^\mathrm{T} x^{( t)}-| {a_i^\mathrm{T} x^{( t)}}|=0\ge p_i^{( t)} a_i^\mathrm{T} x^{( {t+1})}-| {a_i^\mathrm{T} x^{( {t+1})}}|. \end{aligned}$$

(19)

This inequality holds for each i; thus by adding them together, we obtain

$$\begin{aligned} \sum {\Big ( {p_i^{( t)} a_i^\mathrm{T} x^{( t)}-| {a_i^\mathrm{T} x^{( t)}}|})} \ge \sum {( {p_i^{( t)} a_i^\mathrm{T} x^{( {t+1})}-| {a_i^\mathrm{T} x^{( {t+1})}}|}\Big )}. \end{aligned}$$

(20)

Adding (20) to (18), we arrive at

$$\begin{aligned} f( {x^{( t)}})= & {} g( {x^{( t)}})-\sum {| {a_i^\mathrm{T} x^{( t)}}|} \ge g( {x^{( {t+1})}})-\sum {| {a_i^\mathrm{T} x^{( {t+1})}}|}\nonumber \\= & {} f( {x^{( {t+1})}}). \end{aligned}$$

(21)

In other words, for two consecutive iterations $x^{( t)}$ and $x^{( {t+1})}$ of Algorithm 1, the value of the original objective f( x) in (12) or (14) will not increase. Considering the objective function (12) is bounded from below on the feasible region, the established monotone convergence theorem can guarantee that Algorithm 1 will converge.

Second, suppose that Algorithm 1 converges to a limit point $x^*$, according to subproblem (15), $x^*$ should satisfy the following problem:

$$\begin{aligned}&x^*=\arg \min g( x)-\sum {sgn( {a_i^\mathrm{T} x^*})a_i^\mathrm{T} x}\nonumber \\&\text{ s.t. } M^\mathrm{T}x=0,x^\mathrm{T}x=1. \end{aligned}$$

(22)

Then, the Lagrangian function of (22) can be written as

$$\begin{aligned} L=g( x)-\sum {sgn( {a_i^\mathrm{T} x^*})a_i^\mathrm{T} x} -\lambda _1^\mathrm{T} M^\mathrm{T}x-\lambda _2 ( {x^\mathrm{T}x-1}), \end{aligned}$$

(23)

where $\lambda _1$,$\lambda _2$ are the Lagrangian multiplier vectors. Since $x^*$ is the optimal solution of (22), we can obtain the following equation by following the KKT condition [30]

$$\begin{aligned} {\frac{\partial L}{\partial x}}|_{x=x^*} =\nabla g( {x^*})-\sum {sgn( {a_i^\mathrm{T} x^*})x^*} -M\lambda _1 -2\lambda _2 x^*=0. \end{aligned}$$

(24)

The above equation is exactly the KKT condition of the original problem (12), thus showing that $x^*$ is a local optimal solution of (12).

Appendix 2: Proof of Proposition 2

Proof

To solve constrained minimization problem (16), we introduce the following Lagrangian function:

$$\begin{aligned} L( {x,\alpha ,\beta })=\frac{1}{2}\Vert {y-x}\Vert ^2+\alpha ^\mathrm{T}M^\mathrm{T}x+\beta ( {x^\mathrm{T}x-1}), \end{aligned}$$

(25)

where $\alpha$ and $\beta$ are vectors of Lagrangian multipliers. According to the KKT conditions [30], we obtain the following equation:

$$\begin{aligned} \frac{\partial L}{\partial x}=x-y+M\alpha +2\beta x=0\rightarrow x=\frac{y-M\alpha }{1+2\beta }. \end{aligned}$$

(26)

Substituting (26) into (25) yields the following dual problem in variables $\alpha$ and $\beta$:

$$\begin{aligned} \max L( {\alpha ,\beta })=-\frac{1}{2( {1+2\beta })}( {y-M\alpha })^\mathrm{T}( {y-M\alpha })-\beta . \end{aligned}$$

(27)

Noting that the above maximization problem is unconstrained in $\alpha$ and $\beta$, it follows that

$$\begin{aligned} \frac{\partial L}{\partial \alpha }=-\frac{1}{2( {1+2\beta })}( {2M^\mathrm{T}M\alpha -2M^\mathrm{T}y})=0, \end{aligned}$$

(28)

$$\begin{aligned} \frac{\partial L}{\partial \beta }=\frac{1}{( {1+2\beta })^2}( {y-M\alpha })^\mathrm{T}( {y-M\alpha })-1=0. \end{aligned}$$

(29)

Solving the above equations and taking into account $M^\mathrm{T}M=I$, we have

$$\begin{aligned} \alpha =M^\mathrm{T}y \mathrm{and} \beta =\frac{1}{2}( {\Vert {y-M\alpha }\Vert -1}). \end{aligned}$$

(30)

Inserting (30) into (26) and considering that $I-MM^\mathrm{T}$ must be positive semi-definite, we finally get

$$\begin{aligned} x=\frac{( {I-MM^\mathrm{T}})y}{\Vert {( {I-MM^\mathrm{T}})y}\Vert }. \end{aligned}$$

(31)