A novel high-speed neural model for fast pattern recognition

Abstract

Neural networks have shown good results for detecting a certain pattern in a given image. In this paper, faster neural networks for pattern detection are presented. Such processors are designed based on cross-correlation in the frequency domain between the input matrix and the input weights of neural networks. This approach is developed to reduce the computation steps required by these faster neural networks for the searching process. The principle of divide and conquer strategy is applied through matrix decomposition. Each matrix is divided into smaller in size sub-matrices and then each one is tested separately using a single faster neural processor. Furthermore, faster pattern detection is obtained using parallel processing techniques to test the resulting submatrices at the same time using the same number of faster neural networks. In contrast to faster neural networks, the speed up ratio is increased with the size of the input matrix when using faster neural networks and matrix decomposition. Moreover, the problem of local sub-matrix normalization in the frequency domain is solved. The effect of matrix normalization on the speed up ratio of pattern detection is discussed. Simulation results show that local sub-matrix normalization through weight normalization is faster than sub-matrix normalization in the spatial domain. The overall speed up ratio of the detection process is increased as the normalization of weights is done off line.

Appendices

Appendix 1

1.1 An example proves that the cross-correlation between any two matrices is not commutative

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right] $$

Then, the cross-correlation between X and W can be obtained as follows:

$$ \begin{gathered} W \otimes X = \left[ {\begin{array}{*{20}c} 6 &\quad 5 \\ 9 &\quad 8 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 5 &\quad 1 \\ 3 &\quad 7 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {8 \times 5} & {8 \times 1 + 9 \times 5} & {9 \times 1} \\ {5 \times 5 + 8 \times 3} &\quad {6 \times 5 + 5 \times 1 + 9 \times 3 + 8 \times 7} &\quad {6 \times 1 + 9 \times 7} \\ {5 \times 3} & {6 \times 3 + 5 \times 7} & {6 \times 7} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {40} &\quad {53} &\quad 9 \\ {49} &\quad {118} &\quad {69} \\ {15} &\quad {53} &\quad {42} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

On the other hand, the cross-correlation between W and X can be computed as follows:

$$ \begin{gathered} X \otimes W = \left[ {\begin{array}{*{20}c} 5 &\quad 1 \\ 3 &\quad 7 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 6 &\quad 5 \\ 9 &\quad 8 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {7 \times 6} & {3 \times 6 + 7 \times 5} & {3 \times 5} \\ {1 \times 6 + 7 \times 9} &\quad {5 \times 6 + 1 \times 5 + 3 \times 9 + 7 \times 8} &\quad {5 \times 5 + 3 \times 8} \\ {1 \times 9} & {5 \times 9 + 1 \times 8} & {5 \times 8} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {42} &\quad {53} &\quad {15} \\ {69} &\quad {118} &\quad {49} \\ 9 &\quad {53} &\quad {40} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

which proves that X ⊗ W ≠ W ⊗ X.

Also, when one of the two matrices is symmetric, the cross-correlation between the two matrices is non-commutative as shown in the following example:

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 3 \\ 3 & 5 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right] $$

Then, the cross-correlation between X and W can be obtained as follows:

$$ \begin{gathered} X \otimes W = \left[ {\begin{array}{*{20}c} 5 &\quad 3 \\ 3 &\quad 5 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 6 &\quad 5 \\ 9 &\quad 8 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {5 \times 6} & {3 \times 6 + 5 \times 5} & {3 \times 5} \\ {3 \times 6 + 5 \times 9} &\quad {5 \times 6 + 3 \times 5 + 3 \times 9 + 5 \times 8} &\quad {5 \times 5 + 3 \times 8} \\ {3 \times 9} & {5 \times 9 + 3 \times 8} & {5 \times 8} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {30} &\quad {43} &\quad {15} \\ {63} &\quad {112} &\quad {49} \\ {27} &\quad {69} &\quad {40} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

On the other hand, the cross-correlation between W and X can be computed as follows:

$$ \begin{gathered} W \otimes X = \left[ {\begin{array}{*{20}c} 6 &\quad 5 \\ 9 &\quad 8 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 5 &\quad 3 \\ 3 &\quad 5 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {8 \times 5} & {8 \times 3 + 9 \times 5} & {9 \times 3} \\ {5 \times 5 + 8 \times 3} &\quad {6 \times 5 + 5 \times 3 + 9 \times 3 + 8 \times 5} &\quad {6 \times 3 + 9 \times 5} \\ {5 \times 3} & {6 \times 3 + 5 \times 5} & {6 \times 5} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {40} &\quad {69} &\quad {27} \\ {49} &\quad {112} &\quad {63} \\ {15} &\quad {43} &\quad {30} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

which proves that X ⊗ W ≠ W ⊗ X.

The cross-correlation between any two matrices is commutative only when the two matrices are symmetric as shown in the following example.

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 3 \\ 3 & 5 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 8 & 9 \\ 9 & 8 \\ \end{array} } \right] $$

Then, the cross-correlation between X and W can be obtained as follows:

$$ \begin{gathered} X \otimes W = \left[ {\begin{array}{*{20}c} 5 &\quad 3 \\ 3 &\quad 5 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 8 &\quad 9 \\ 9 &\quad 8 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {5 \times 8} & {9 \times 5 + 8 \times 3} & {9 \times 3} \\ {9 \times 5 + 8 \times 3} &\quad {8 \times 5 + 9 \times 3 + 9 \times 3 + 8 \times 5} &\quad {8 \times 3 + 9 \times 5} \\ {9 \times 3} & {8 \times 3 + 9 \times 5} & {8 \times 5} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {40} &\quad {69} &\quad {27} \\ {69} &\quad {122} &\quad {69} \\ {27} &\quad {69} &\quad {40} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

On the other hand, the cross-correlation between W and X can be computed as follows:

$$ \begin{gathered} W \otimes X = \left[ {\begin{array}{*{20}c} 8 &\quad 9 \\ 9 &\quad 8 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 5 &\quad 3 \\ 3 &\quad 5 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {8 \times 5} & {9 \times 5 + 8 \times 3} & {9 \times 3} \\ {9 \times 5 + 8 \times 3} &\quad {8 \times 5 + 9 \times 3 + 9 \times 3 + 8 \times 5} &\quad {9 \times 5 + 8 \times 3} \\ {9 \times 3} & {5 \times 9 + 3 \times 8} & {8 \times 5} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {40} &\quad {69} &\quad {27} \\ {69} &\quad {122} &\quad {69} \\ {27} &\quad {69} &\quad {40} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

which proves that the cross-correlation is commutative (X ⊗ W = W ⊗ X) only under the condition when the two matrices X and W are symmetric.

Appendix 2

2.1 An example proves that the cross-correlation between any two matrices is different from their convolution

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right], $$

the result of their cross-correlation can be computed as illustrated from the previous example (first result) in Appendix 1. The convolution between W and X can be obtained as follows:

$$ \begin{gathered} W\diamondsuit X = \left[ {\begin{array}{*{20}c} 8 &\quad 9 \\ 5 &\quad 6 \\ \end{array} } \right]\diamondsuit \left[ {\begin{array}{*{20}c} 5 &\quad 1 \\ 3 &\quad 7 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {6 \times 5} & {5 \times 5 + 6 \times 1} & {5 \times 1} \\ {9 \times 5 + 6 \times 3} &\quad {8 \times 5 + 9 \times 1 + 5 \times 3 + 6 \times 7} &\quad {8 \times 1 + 5 \times 7} \\ {9 \times 3} & {8 \times 3 + 9 \times 7} & {8 \times 7} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {30} &\quad {31} &\quad 5 \\ {63} &\quad {106} &\quad {43} \\ {27} &\quad {87} &\quad {56} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

which proves that W ⊗ X ≠ W⋄X.

When the second matrix W is symmetric, the cross-correlation between W and X can be computed as follows:

$$ \begin{gathered} W \otimes X = \left[ {\begin{array}{*{20}c} 8 &\quad 9 \\ 9 &\quad 8 \\ \end{array} } \right] \otimes \left[ {\begin{array}{*{20}c} 5 &\quad 1 \\ 3 &\quad 7 \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {8 \times 5} & {9 \times 5 + 8 \times 1} & {9 \times 1} \\ {9 \times 5 + 8 \times 3} &\quad {8 \times 5 + 9 \times 3 + 9 \times 1 + 8 \times 7} &\quad {8 \times 1 + 7 \times 9} \\ {9 \times 3} & {8 \times 3 + 9 \times 7} & {8 \times 7} \\ \end{array} } \right] \hfill \\ = \left[ {\begin{array}{*{20}c} {40} &\quad {87} &\quad 9 \\ {79} &\quad {106} &\quad {71} \\ {45} &\quad {53} &\quad {56} \\ \end{array} } \right] \hfill \\ \end{gathered} $$

while the convolution can be between W and X can be obtained as follows:

$$ \begin{aligned} W\diamondsuit X & = \left[ {\begin{array}{*{20}c} 8 &\quad 9 \\ 9 &\quad 8 \\ \end{array} } \right]\diamondsuit \left[ {\begin{array}{*{20}c} 5 &\quad 1 \\ 3 &\quad 7 \\ \end{array} } \right] \\ & = \left[ {\begin{array}{*{20}c} {8 \times 5} & {9 \times 5 + 8 \times 1} & {9 \times 1} \\ {9 \times 5 + 8 \times 3} &\quad {8 \times 5 + 9 \times 3 + 9 \times 1 + 8 \times 7} &\quad {8 \times 1 + 7 \times 9} \\ {9 \times 3} & {8 \times 3 + 9 \times 7} & {8 \times 7} \\ \end{array} } \right] \\ & = \left[ {\begin{array}{*{20}c} {40} &\quad {87} &\quad 9 \\ {79} &\quad {106} &\quad {71} \\ {45} &\quad {53} &\quad {56} \\ \end{array} } \right] \\ \end{aligned} $$

which proves that under the condition that the second matrix is symmetric (or the two matrices are symmetric) the cross-correlation between any the two matrices equals to their convolution.

Appendix 3

3.1 A cross-correlation example between a normalized matrix and other non-normalized one and vise versa

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right] $$

Then the normalized matrices $\overline{X} ,$ and $\overline{W}$ can be computed as:

$$ \overline{X} = \left[ {\begin{array}{*{20}c} 1 & { - 3} \\ { - 1} & 3 \\ \end{array} } \right],\quad {\rm and}\quad \overline{W} = \left[ {\begin{array}{*{20}c} { - 1} & { - 2} \\ 2 & 1 \\ \end{array} } \right]. $$

Now, the cross-correlation between a normalized matrix and the other non-normalized one can be computed as follows:

$$ \overline{X} \otimes W = \left[ {\begin{array}{*{20}c} 1 & { - 3} \\ { - 1} & 3 \\ \end{array} } \right]\;\left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {18} & 9 & { - 5} \\ 9 & 6 & { - 3} \\ { - 27} & { - 15} & 8 \\ \end{array} } \right] $$

$$ X \otimes \overline{W} = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right]\;\left[ {\begin{array}{*{20}c} { - 1} & { - 2} \\ 2 & 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} { - 7} & { - 17} & { - 6} \\ {13} & 6 & { - 7} \\ 2 & {11} & 5 \\ \end{array} } \right] $$

which means that $\overline{X} \otimes W \ne X \otimes \overline{W}$.

However, the two results are equal only at the center element which equals to the dot product between the two matrices. The value of the center element (2, 2) = 6 as shown above and also in Appendix 4.

Appendix 4

4.1 A dot product example between a normalized matrix and other non-normalized one and vise versa

This is to validate the correctness of Eq. 51. The left hand side of Eq. 51 can be expressed as follows:

$$ \overline{X} \cdot W = \left[ \begin{gathered} X_{1,1} - \overline{X} \ldots X_{1,n} - \overline{X} \hfill \\ \vdots\\ X_{n,1} - \overline{X} \ldots X_{n,n} - \overline{X} \hfill \\ \end{gathered} \right] \cdot \left[ \begin{gathered} W_{1,1} \ldots W_{1,n} \hfill \\ \vdots\\ W_{n,1} \ldots W_{n,n} \hfill \\ \end{gathered} \right] $$

(58)

and also the right hand side of the same can be represented as:

$$ X \cdot \overline{W} = \left[ \begin{gathered} X_{1,1} \ldots X_{1,n} \hfill \\ \vdots \\ X_{n,1} \ldots X_{n,n} \hfill \\ \end{gathered} \right] \cdot \left[ \begin{gathered} W_{1,1} - \overline{W} \ldots W_{1,n} - \overline{W} \hfill \\ \vdots\\ W_{n,1} - \overline{W} \ldots W_{n,n} - \overline{W} \hfill \\ \end{gathered} \right]. $$

(59)

$\overline{X}$ and $\overline{W}$ are defined as follows:

$$ \begin{aligned} \overline{X} &= {\frac{{X_{1,1} + X_{1,2} + \cdots + X_{n,n} }}{{n^{2} }}} \\ \overline{W} &= {\frac{{W_{1,1} + W_{1,2} + \cdots + W_{n,n} }}{{n^{2} }}} \\ \end{aligned} $$

(60)

By substituting from Eq. 60 in Eqs. 58 and 59, then simplifying the results we can easily conclude that $\overline{X}_{rc} W_{i} = X_{rc} \overline{W}_{i} .$

Here is also a practical example:

$$ {\text{Let}}\quad X = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right],\quad {\text{and}}\quad W = \left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right]. $$

Then the normalized matrices $\overline{X},$ and $\overline{W}$ can be computed as:

$$ \overline{X} = \left[ {\begin{array}{*{20}c} 1 & { - 3} \\ { - 1} & 3 \\ \end{array} } \right],\quad {\rm and}\quad \overline{W} = \left[ {\begin{array}{*{20}c} { - 1} & { - 2} \\ 2 & 1 \\ \end{array} } \right]. $$

Now, the dot product between a normalized matrix and the other non-normalized one can be performed as follows:

$$ \overline{X} \cdot W = \left[ {\begin{array}{*{20}c} 1 & { - 3} \\ { - 1} & 3 \\ \end{array} } \right]\;\left[ {\begin{array}{*{20}c} 6 & 5 \\ 9 & 8 \\ \end{array} } \right] = 6 - 15 - 9 + 24 = 6 $$

$$ X \cdot \overline{W} = \left[ {\begin{array}{*{20}c} 5 & 1 \\ 3 & 7 \\ \end{array} } \right]\;\left[ {\begin{array}{*{20}c} { - 1} & { - 2} \\ 2 & 1 \\ \end{array} } \right] = - 5 - 2 + 6 + 7 = 6 $$

which means generally that the dot product between a normalized matrix X and non-normalized matrix W equals to the dot product between the normalized matrix W and non-normalized matrix X. On the other hand, the cross-correlation results are different as proved in Appendix 3.