Novel hybrid SVM-TLBO forecasting model incorporating dimensionality reduction techniques

Abstract

In this paper, we present a highly accurate forecasting method that supports improved investment decisions. The proposed method extends the novel hybrid SVM-TLBO model consisting of a support vector machine (SVM) and a teaching-learning-based optimization (TLBO) method that determines the optimal SVM parameters, by combining it with dimensional reduction techniques (DR-SVM-TLBO). The dimension reduction techniques (feature extraction approach) extract critical, non-collinear, relevant, and de-noised information from the input variables (features), and reduce the time complexity. We investigated three different feature extraction techniques: principal component analysis, kernel principal component analysis, and independent component analysis. The feasibility and effectiveness of this proposed ensemble model were examined using a case study, predicting the daily closing prices of the COMDEX commodity futures index traded in the Multi Commodity Exchange of India Limited. In this study, we assessed the performance of the new ensemble model with the three feature extraction techniques, using different performance metrics and statistical measures. We compared our results with results from a standard SVM model and an SVM-TLBO hybrid model. Our experimental results show that the new ensemble model is viable and effective, and provides better predictions. This proposed model can provide technical support for better financial investment decisions and can be used as an alternative model for forecasting tasks that require more accurate predictions.

Appendices

Appendix A:: Technical indicators (features) used in this study

ᅟ ᅟ

Appendix B:: Dimensionality reduction techniques used in this study

The objective of a dimension reduction technique is to reduce the dimension (number of features) of the input from a high-dimensional space to a low dimensional subspace. Dimensional reduction methods can be divided into two types: (i) feature selection and (ii) feature extraction. In feature selection, a subset of features is selected from the originals. In feature extraction new features are computed by transforming the original features. We present brief reviews of the dimensional reduction methods based on feature extraction that were used in our study. That is, PCA, KPCA, and ICA.

B.1 Principal component analysis (PCA)

Principal component analysis is a well-known linear statistical approach for feature extraction. The objective is to reduce the dimension of the input features from the original dataset [20]. It uses an orthogonal transformation to convert a set of N patterns (samples) of l possibly correlated features into a set of Nsamples of m(≤l) uncorrelated features called principal components (PCs). The transformation mechanism is designed such that the first principal component (PC1) has the highest possible variance, the second principal component (PC2) is orthogonal to the PC1 and accounts for next highest variance, and so on for the other PCs.

The PCA procedure is briefly described as follows

Step 1::

Input Npatterns (samples) X ₁,X ₂,…,X _N that each have l features (X _j ∈R ^l). Each vector X _j for j=1,2,…,N is such that the mean value of the features in X _j is zero (that is, we subtract the mean value of the original feature from each feature value).

Step 2::

Compute the covariance matrix

$$ C=\frac{1}{N}\sum\limits_{k=1}^{N} {X_{k} {X_{k}^{T}}} $$

(B.1)

The ij-th element of matrix Cis

$$ C_{ij} =\frac{1}{N}\sum\limits_{k=1}^{N} {X_{k} (i)X_{k} (j)} $$

(B.2)

where X _k (i)denotes the ith component of the X _k sample.

Step 3::

Calculate l eigenvalues of C and arrange them in non-increasing order λ ₁≥λ ₂≥...≥λ _l . For each eigenvalue λ _i , i=1,2,…,l, compute an associated eigenvector α _i ∈R ^lof matrix C using an eigenvector decomposition technique [35].

Step 4::

Choose the m≤l largest eigenvalues (choose the smallest integer m, so that λ _m−1−λ _m is large or $\sum \limits _{i=1}^{m} {\lambda _{i} } \ge t\sum \limits _{i=1}^{N} {\lambda _{i} }$ where t=0.95 if we wish to retain 95 % variance in the transformed data, where $\sum \limits _{i=1}^{N} {\lambda _{i} }$ represents the total variance).

Step 5::

Use the eigenvectors (column vectors) α ₁,α ₂,…,α _m to form the transformation matrix.

$$ A=[\alpha_{1} \alpha_{2} ...\alpha_{m}] $$

(B.3)

Step 6::

Transform each pattern X _i in the original space R ^l to the vector Y _i in the m-dimensional space R ^m(m<l) using

$$ Y_{i} =A^{T}X_{i} ,i=1,2,\ldots,N $$

(B.4)

So the jth component Y _i (j) of Y _i is the projection of X _i on α _i (i.e., $Y_{i} (j)=\alpha _{j}^{T} X_{i} )$.

B.2 Kernel principal component analysis (KPCA)

In the PCA technique, each input pattern (sample) in R ^l is linearly projected onto a lower dimensional subspace. This is appropriate when the data approximately lie on a linear manifold (for example a hyperplane). However, in many applications the input data lie on a low dimensional nonlinear manifold. Then it is more appropriate to use KPCA, which is a nonlinear dimensional reduction technique. In this method the input patterns X _i ∈R ^l for i=1,2,…,N (where N is the number of input samples) are first mapped onto a space H with more than l dimensions using a non-linear mapping ϕ:R ^l→H[42]. Their images ϕ(X _i ) are projected along the orthonormal eigenvectors of the covariance matrix of ϕ(X _i )’s. These projections only involve the inner product of the ϕ(X _i )’s in H,ϕ is not explicitly known, and it is difficult to construct a kernel function. So we use Kdefined by K:R ^l×R ^l→R such that

$$ K(X_{i} ,X_{j} )=<\phi (X_{i} ),\phi (X_{j})> $$

(B.5)

(where <, > denotes inner product in H) to compute the inner products involved in the projections leading to the computation of Y _i ’s having fewer dimensions m(m<l)than X _i ’s. It has been proved that the components Y _i (k),k=1,2,…,m of the Y _i ’s are uncorrelated and the first q(≤m) principal components have maximum mutual information with respect to the inputs, which justifies the use of the method for dimensionality reduction.

The KPCA procedure is given in the form of the following algorithm.

Step 1::

Input the data patterns (samples) X _i ∈R ^l for i=1,2,…,N (where N is the number of input samples).

Step 2::

Choose a kernel function K:R ^l×R ^l→R and compute the kernel matrix K ₁whose ij-th element is equal to K(X _i ,X _j ) for i,j=1,2,…,l

Step 3::

Compute the eigenvalues and eigenvectors of K ₁. Arrange the eigenvalues in non-increasing order λ ₁≥λ ₂≥...≥λ _l . Let the corresponding eigenvectors be a ₁,a ₂,…,a _l .

Step 4::

Choose mdominant eigenvalues λ ₁,λ ₂,…,λ _m (m≤l)[choose the smallest integer m such that λ _m−1−λ _m is large or $\sum \limits _{i=1}^{m} {\lambda _{i} } \ge t\sum \limits _{i=1}^{N} {\lambda _{i} }$, where t=0.95 if we wish to retain 95% of the variance in the transformed data, and $\sum \limits _{i=1}^{N} {\lambda _{i} }$represents the total variance], and normalize the corresponding eigenvectors a ₁,a ₂,…,a _m using

$$ a_{k}^{\prime} =\frac{a_{k} }{\left\| {a_{k} } \right\|\sqrt {\lambda_{k} } },k=1,2,\ldots,m $$

(B.6)

Step 5::

For each X _i ,i=1,2,…,N, compute the m projections Y _i (k) of ϕ(X _i ) onto each of the orthonormal eigenvectors $a_{{k}^{\prime }}$’s, k=1,2,…,m,i.e.,

$$ Y_{i} (k)=\sum\limits_{j=1}^{l} {{a_{k}^{i}} (j)K(X_{i} ,X_{j} ),k=} 1,2,\ldots,m $$

(B.7)

B.3 Independent component analysis (ICA)

Independent component analysis (ICA) is a relatively new statistical method [14,16]. ICA does not transform uncorrelated components or factors, but instead attempts to find statistically independent components or factors in the transformed vectors. The primary goal of this method is to find representations of non-Gaussian data, so those components are statistically independent or as independent as possible [16].

In ICA, we assume that lmeasured variables X=[x ₁,x ₂,…,x _l ]^T can be expressed as linear combinations of n unknown latent source components S=[s ₁,s ₂,…,s _n ]^T, i.e.,

$$ X=AS $$

(B.8)

where A _l×l is an unknown mixing matrix. Here, we consider that l≥n if A is a full rank matrix. S is the latent source data that cannot be directly observed from the input mixture data, X. The basic ICA objective is to estimate the latent source components, S, and unknown mixing matrix A from X with appropriate assumptions on the statistical properties of the source distribution. The basic ICA model for feature transformation aims to find a de-mixing matrix W _l×l that can be written as

$$ Y=WX $$

(B.9)

where Y=[y ₁,y ₂,…,y _n ]^T is the independent component vector. The elements of Ymust be statistically independent and are called independent components (ICs). Here, W = A ⁻¹(i.e., the de-mixing matrix Wis the inverse of mixing matrix A). The ICs (y _i ) can be used to compute the latent source signals s _i .

Many algorithms can perform the ICA. The fixed-point fast ICA method presented by Hyvärinen and Oja [15] is the most popular. We used fixed-point fast ICA in our experimental study. In this algorithm, PCA is first used to transform the original input vectors (X) to a set of new uncorrelated vectors with zero means and unity variance. This process reduces the dimension of X and consequently reduces the number of Y. Then, the uncorrelated vector obtained by PCA is used to estimate the independent components vectors (Y) and the transformed matrix using the fixed point algorithm.