Spatial interest pixels (SIPs): useful low-level features of visual media data

Abstract

Visual media data such as an image is the raw data representation for many important applications. Reducing the dimensionality of raw visual media data is desirable since high dimensionality degrades not only the effectiveness but also the efficiency of visual recognition algorithms. We present a comparative study on spatial interest pixels (SIPs), including eight-way (a novel SIP detector), Harris, and Lucas‐Kanade, whose extraction is considered as an important step in reducing the dimensionality of visual media data. With extensive case studies, we have shown the usefulness of SIPs as low-level features of visual media data. A class-preserving dimension reduction algorithm (using GSVD) is applied to further reduce the dimension of feature vectors based on SIPs. The experiments showed its superiority over PCA.

Appendix

1.1 Generalized discriminant analysis using GSVD

In this Appendix, we will first complete the formulation of the optimization problem (5.10), and then give the proof of Theorem 5.1.

From Eq. 5.9, we have

$$\begin{array}{*{20}c} {X^{T} H_{b} H^{T}_{b} X = {\left[ {\begin{array}{*{20}l} {{{\sum\nolimits_1^T {} }} \hfill} \\ {0 \hfill} \\ \end{array} } \right]}U^{T} U{\left[ {{\sum\nolimits_1 {\quad 0} }} \right]}} \\ { = {\left[ {\begin{array}{*{20}l} {{{\sum\nolimits_1^T {{\sum\nolimits_1 {} }} }} \hfill} & {0 \hfill} \\ {0 \hfill} & {0 \hfill} \\ \end{array} } \right]} \equiv D_{1} ,} \\ {X^{T} H_{w} H^{T}_{w} X = {\left[ {\begin{array}{*{20}l} {{{\sum\nolimits_2^T {} }} \hfill} \\ {0 \hfill} \\ \end{array} } \right]}V^{T} V{\left[ {{\sum\nolimits_2 {\quad 0} }} \right]}} \\ { = {\left[ {\begin{array}{*{20}l} {{{\sum\nolimits_2^T {} }} \hfill} & {0 \hfill} \\ {0 \hfill} & {0 \hfill} \\ \end{array} } \right]} \equiv D_{2} .} \\ \end{array} $$

Hence

$$\begin{array}{*{20}c} {S^{L}_{b} = G^{T} S_{b} G = G^{T} H_{b} H^{T}_{b} G = \widetilde{G}D_{1} \widetilde{G}^{T} ,} \\ {S^{L}_{w} = G^{T} S_{w} G = G^{T} H_{w} H^{T}_{w} G = \widetilde{G}D_{2} \widetilde{G}^{T} ,} \\ \end{array} $$

(A.11)

where the matrix $\tilde{G} =(X^{-1} G)^T$.

We will use the above representations for $S_b^L$ and $S_w^L$ in Section A for the minimizations of $F$.

We first formulate the optimization problem in Eq. 5.8 as the following:

$$\begin{aligned} & \min {\text{imize}}\quad {\text{trace}}{\left( {S^{L}_{w} } \right)} \\ & {\text{subject to }}\;{\text{trace}}{\left( {S^{L}_{b} } \right)} = 1. \\ \end{aligned} $$

(A.12)

Recall by Eq. A.11,

$$\begin{array}{*{20}c} {{\text{trace}}{\left( {S^{L}_{b} } \right)} = {\text{trace}}{\left( {\widetilde{{\text{G}}}D_{1} \widetilde{{\text{G}}}^{T} } \right)} = {\text{trace}}{\left( {D_{1} \widetilde{{\text{G}}}^{T} \widetilde{{\text{G}}}} \right)},} \\ {{\text{trace}}{\left( {S^{L}_{w} } \right)}{\text{ = trace}}{\left( {\widetilde{{\text{G}}}D_{2} \widetilde{{\text{G}}}^{T} } \right)} = {\text{trace}}{\left( {D_{2} \widetilde{{\text{G}}}^{T} \widetilde{{\text{G}}}} \right)}.} \\ \end{array} $$

Let $u_{ij}$ be the $ij$-th term of the matrix $\tilde{G}^T \tilde{G}$, then

$${\text{trace}}{\left( {S^{L}_{b} } \right)} = {\sum\limits_{i = 1}^r {u_{{ii}} + } }{\sum\limits_{i = r + 1}^{r + s} {\alpha ^{2}_{i} u_{{ii}} = 1} }$$

(A.13)

$${\text{trace}}{\left( {S^{L}_{w} } \right)} = {\sum\limits_{i = r + 1}^{r + s} {\beta ^{2}_{i} u_{{ii}} + {\sum\limits_{i = r + s + 1}^t {u_{{ii}} } }.} }$$

(A.14)

Since $\alpha_i^2 + \beta_i^2 = 1$, for $r+1 \le i \le r+s$, we have

$$\begin{array}{*{20}c} {{\text{trace}}{\left( {S^{L}_{w} } \right)} + {\text{trace}}{\left( {S^{L}_{b} } \right)}} \\ { = {\sum\limits_{i = 1}^r {u_{{ii}} + {\sum\limits_{i = r + 1}^{r + 2} {{\left( {\alpha ^{2}_{i} + \beta ^{2}_{i} } \right)}u_{{ii}} {\sum\limits_{i = r + s + 1}^t {u_{{_{{ii}} }} } }} }} }} \\ { = {\sum\limits_{i = 1}^t {u_{{ii}} ,} }} \\ \end{array} $$

hence $\mbox{trace} ( S_w^L)=\sum_{i=1}^t u_{ii}-\mbox{trace} ( S_b^L)=\sum_{i=1}^t u_{ii} -1$.

Therefore the original optimization (5.8) is equivalent to the following

$$\begin{array}{*{20}c} {\min {\text{imize trace}}{\left( {S^{L}_{w} } \right)} = {\sum\limits_{i = 1}^t {u_{{ii}} - 1} }} \\ {{\text{subject to trace}}{\left( {S^{L}_{b} } \right)} = {\sum\limits_{i = 1}^r {u_{{ii}} + {\sum\limits_{i = r + 1}^{r + 2} {\alpha ^{2}_{i} u_{{ii}} = 1} }.} }} \\ \end{array} $$

(A.15)

Now we begin the proof of Theorem 5.1. First, note that $u_{ii}$'s are diagonal elements of a positive semi-definite matrix, hence nonnegative. Since $u_{ii}$ is the diagonal element of a positive semi-definite matrix $\tilde{G}^T \tilde{G}$, if $u_{ii} = 0$ for some $i$, then $u_{ij} = u_{ji}= 0$ for every $j$.

Recall the matrix $\tilde{G}^T \tilde{G}$ is an $m$ by $m$ matrix with $m$ diagonal entries. However only the first $t$ diagonal entries $\{u_{ii}\}_{i=1}^t$ appear in the optimization problem (5.10), hence the last $m-t$ diagonal entries of the matrix $\tilde{G}^T \tilde{G}$ don't affect the optimization problem (5.10). For simplicity, we set the last $m-t$ diagonal entries of the matrix $\tilde{G}^T \tilde{G}$ to be zero, i.e., $u_{ii}=0$, for $i = t+1, \cdots, m$.

For $\{u_{ii}\}_{i=r+s+1}^t$, any positive value for $u_{ii}$, when $r+s+1 \le i \le t$ would increase the objective function in Eq. 5.10, while keeping the constraint unchanged. Hence we have $u_{ii}=0$, for $r+s+1 \le i \le t$. Thus we obtain Theorem 5.1.