Speeding up the scaled conjugate gradient algorithm and its application in neuro-fuzzy classifier training

Abstract

The aim of this study is to speed up the scaled conjugate gradient (SCG) algorithm by shortening the training time per iteration. The SCG algorithm, which is a supervised learning algorithm for network-based methods, is generally used to solve large-scale problems. It is well known that SCG computes the second-order information from the two first-order gradients of the parameters by using all the training datasets. In this case, the computation cost of the SCG algorithm per iteration is more expensive for large-scale problems. In this study, one of the first-order gradients is estimated from the previously calculated gradients without using the training dataset. To estimate this gradient, a least square error estimator is applied. The estimation complexity of the gradient is much smaller than the computation complexity of the gradient for large-scale problems, because the gradient estimation is independent of the size of dataset. The proposed algorithm is applied to the neuro-fuzzy classifier and the neural network training. The theoretical basis for the algorithm is provided, and its performance is illustrated by its application to several examples in which it is compared with several training algorithms and well-known datasets. The empirical results indicate that the proposed algorithm is quicker per iteration time than the SCG. The algorithm decreases the training time by 20–50% compared to SCG; moreover, the convergence rate of the proposed algorithm is similar to SCG.

Appendix: The presentation of the complexity of the gradient estimation

Our claim is that the complexity of the gradient estimation $g_{t,k}^{est}$ is less than the complexity of the gradient calculation $g_{t,k}^{calc} ,$ that is $O\left( {g_{t,k}^{est} } \right) < O\left( {g_{t,k}^{calc} } \right),$ where O(·) is the calculation complexity.

This can be proved with ease:

Both $g_{t,k}^{est}$ and $g_{t,k}^{calc}$ represent the gradient of E with respect to ρ _ij .

Calculation of the operation size of $g_{t,k}^{est} ;$

$$ {\text{If}}\,{\mathbf{A}} = \left[ {\begin{array}{*{20}c} {\theta_{k - 2}^{2} } & {\theta_{k - 2} } & 1 \\ {\theta_{k - 1}^{2} } & {\theta_{k - 1} } & 1 \\ {\theta_{k}^{2} } & {\theta_{k} } & 1 \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {{\varvec{\Uptheta}}_{k - 2} } \\ {{\varvec{\Uptheta}}_{k - 1} } \\ {{\varvec{\Uptheta}}_{k} } \\ \end{array} } \right],\quad {\text{then}}\,O({\mathbf{A}}) = 3\,mult $$

(9)

In Eq. 9, the “mult” term represents the operation of multiplication.

$$ {\text{If}}\,{\mathbf{F}} = \left[ {\begin{array}{*{20}c} {f_{1} } \\ {f_{2} } \\ {f_{3} } \\ \end{array} } \right]\quad {\text{and}}\quad {\mathbf{G}} = \left[ {\begin{array}{*{20}c} {g_{k - 2} } \\ {g_{k - 1} } \\ {g_{k} } \\ \end{array} } \right] \Rightarrow {\mathbf{AF}} = {\mathbf{G}} \Rightarrow {\mathbf{F}} = {\mathbf{A}}^{ - 1} {\mathbf{G}}. $$

(10)

$g_{t,k}^{est}$ is estimated by LSE as shown below:

$$ g_{t,k}^{est} = {\varvec{\Uptheta}}_{t,k} {\mathbf{F}} = f_{1} \theta_{t,k}^{2} + f_{2} \theta_{t,k} + f_{3} . $$

(11)

If ${\mathbf{A}} = \left[ {\begin{array}{*{20}c} {a_{11} } & {a_{12} } & {a_{13} } \\ {a_{21} } & {a_{22} } & {a_{23} } \\ {a_{31} } & {a_{32} } & {a_{33} } \\ \end{array} } \right] ,$ then the determinant of A is calculated as follows:

$$ \left| {\mathbf{A}} \right| = a_{11} a_{22} a_{33} + a_{12} a_{23} a_{31} + a_{13} a_{21} a_{32} - a_{11} a_{23} a_{32} - a_{12} a_{21} a_{33} - a_{13} a_{22} a_{31} . $$

(12)

In addition $O\left( {\left| {\mathbf{A}} \right|} \right) = 5\,add + 2 \times 6\,mult,$ where the term “add” represents the operation of the addition. The adjoint of A is given below:

$$ adj\left( {\mathbf{A}} \right) = \left\{ {\begin{array}{*{20}c} {{\mathbf{A}}_{11} = \left( { - 1} \right)^{1 + 1} = \left| {\begin{array}{*{20}c} {a_{22} } & {a_{23} } \\ {a_{32} } & {a_{33} } \\ \end{array} } \right| = a_{22} a_{33} - a_{23} a_{32} } \\ \vdots \\ {{\mathbf{A}}_{33} = \left( { - 1} \right)^{1 + 1} = \left| {\begin{array}{*{20}c} {a_{11} } & {a_{12} } \\ {a_{21} } & {a_{22} } \\ \end{array} } \right| = a_{11} a_{22} - a_{21} a_{12} } \\ \end{array} } \right.. $$

(13)

The operation size of adj(A) is $O\left( {adj\left( {\mathbf{A}} \right)} \right) = 9 \times \left( {1\,add + 2\,mult} \right).$ Now, A ⁻¹ and its size can be determined as;

$$ {\mathbf{A}}^{ - 1} = \frac{1}{{\det \left( {\mathbf{A}} \right)}}adj\left( {\mathbf{A}} \right) = {\mathbf{B}},\quad {\text{and}}\quad O\left( {{\mathbf{A}}^{ - 1} } \right) = 14\,add + 39\,mult. $$

(14)

$$ {\mathbf{F}} = {\mathbf{A}}^{ - 1} {\mathbf{G}} = \left[ {\begin{array}{*{20}c} {b_{11} g_{k - 2} + b_{12} g_{k - 1} + b_{13} g_{k} } \\ {b_{21} g_{k - 2} + b_{22} g_{k - 1} + b_{23} g_{k} } \\ {b_{31} g_{k - 2} + b_{32} g_{k - 1} + b_{33} g_{k} } \\ \end{array} } \right]\quad {\text{and}}\quad O\left( {\mathbf{F}} \right) = 6\,add + 9\,mult. $$

(15)

Lastly, $O\left( {g_{t,k}^{est} = {\varvec{\Uptheta}}_{t,k} {\mathbf{F}}} \right) = 2\,add + 2\,mult$ and $Total\,O = 22\,add + 53\,mult.$

On the other hand, the calculation of $g_{t,k}^{calc}$ and its number of arithmetic operations are determined below:

Firstly, the output of NFC should be calculated from the first layer of NFC:

$$ \mu_{ij} = \exp \left( { - 0.5\frac{{\left( {x_{pj} - \rho_{ij} } \right)^{2} }}{{\sigma_{ij}^{2} }}} \right), $$

(16)

where $\rho_{ij} \,\left( {\rho_{ij} \in {\varvec{\Upgamma}}_{m \times n} } \right)$ and $\sigma_{ij} \,\left( {\sigma_{ij} \in {\varvec{\Uplambda}}_{m \times n} } \right)$ are the centre and the width of the Gaussian membership function μ _ij for the ith rule and the jth feature. In Eq. 16, the exp(·) operation is approximately calculated using the Maclaurin Series;

$$ \exp \left( z \right) = \sum\limits_{u = 0}^{\infty } {\frac{{z^{u} }}{u!}} = 1 + z + \frac{{z^{2} }}{2!} + \cdots + \frac{{z^{u} }}{u!} + \cdots $$

(17)

If the upper limit for u = 10, then $O\left( {\exp (z)} \right) = 10\,add + 117\,mult$ and $O\left( {\mu_{ij} } \right) = 11\,add + 123\,mult .$ The output of the second layer is calculated as

$$ rule_{i} = \prod\limits_{j = 1}^{n} {\mu_{ij} } ,\quad {\text{and}}\quad O\left( {rule_{i} } \right) = n \times \left( {11\,add + 123\,mult} \right), $$

(18)

where rule _i represents the firing strength of the ith rule. The output of the third layer is calculated as:

$$ o_{k} = \sum\limits_{i = 1}^{m} {rule_{i} w_{ik} } ,\quad {\text{and}}\quad O(o_{k} ) = m \times n \times \left( {11\,add + 123\,mult} \right) + m \times \left( {add + mult} \right), $$

(19)

where $w_{ik} \,\left( {w_{ik} \in {\mathbf{W}}_{m \times c} } \right)$ represents the weight of the ith rule that belongs to the kth class; m denotes the number of rules. The output of the last layer is calculated as

$$ \begin{aligned} out_{k} & = \frac{{o_{k} }}{{\sum\nolimits_{l = 1}^{c} {o_{l} } }} = \frac{{o_{k} }}{\mathbf T},\quad {\mathbf T} = \sum\limits_{l = 1}^{c} {o_{l} } \quad {\text{and}} \\ O\left( {out_{k} } \right) & = m \times n \times \left( {11\,add + 123\,mult} \right) + \left( {m + c} \right)\,add + \left( {m + 1} \right)\,mult. \\ \end{aligned} $$

(20)

The mean square error function is used as the cost function:

$$ E = \frac{1}{N}\sum\limits_{p = 1}^{N} {E^{p} } ,\quad E^{p} = \frac{1}{2}\sum\limits_{k = 1}^{c} {\left( {y_{pk} - out_{pk} } \right)^{2} } , $$

(21)

where N represents the number of samples; c represents the number of classes; y _pk and out _pk are the target and the actual output of the pth sample and the kth class, respectively.$g_{t,k}^{calc}$ of ρ _ij is calculated using the chain rule:

$$ \frac{\partial E}{{\partial \rho_{ij} }} = \frac{1}{N}\sum\limits_{p = 1}^{N} {\sum\limits_{k = 1}^{c} {\left( {out_{pk} - y_{pk} } \right)\left( {\frac{{1 - out_{pk} }}{\rm T}} \right)w_{ik} rule_{i} \left( {\frac{{x_{pj} - \rho_{ij} }}{{\sigma_{ij}^{2} }}} \right)} } . $$

(22)

The total arithmetic operations of $g_{t,k}^{calc}$ of ρ _ij are given below:

$$ \begin{aligned} O\left( {g_{t,k}^{calc} } \right) & = c \times N \times \left( {m \times n \times \left( {11\,add + 123\,mult} \right) + \left( {m + c} \right)\,add + \left( {m + 1} \right)\,mult} \right) \\ & + \quad c \times N \times (9\,mult + 3\,add) + 2\left( {c \times N} \right)\,add + N \times n \times \left( {11\,add + 123\,mult} \right) \\ O\left( {g_{t,k}^{calc} } \right) & \approx \left( {c \times N \times \left( {5 + 11 \times m \times n} \right) + 11 \times N \times n} \right)\,add \\ & + \quad \left( {c \times N \times \left( {9 + 11 \times m \times n} \right) + 123 \times N \times n} \right)\,mult \\ \end{aligned} $$

(23)

The comparison of the number of arithmetic operations of the gradients is given in Eq. 24.

$$ \begin{aligned} & 22\,add + 53\,mult \ll N \times n \times \left( {\left( {c \times \left( {\frac{5}{n} + 11 \times m} \right) + 11} \right)\,add + \left( {c \times \left( {\frac{9}{n} + 11 \times m} \right) + 123} \right)} \right)\,mult \\ & {\text{If}}\,N > 100,\,m > 1,\,n > 1\;{\text{and}}\;c > 1,\;{\text{then}} \\ & O\left( {g_{t,k}^{est} } \right) \ll O\left( {g_{t,k}^{calc} } \right) \\ \end{aligned} $$

(24)