Informed Weighted Random Projection for Dimension Reduction

Abstract

Dimensionality reduction is a frequent pre-processing step in classification tasks. It helps to improve the accuracy of classification by better representing the dataset and also alleviates the curse of dimensionality by reducing the number of dimensions. Traditional dimensionality reduction techniques such as PCA or Kernel PCA are well known techniques that find a lower dimensional subspace which best represents the higher dimensional dataset. On the other hand, random projection can also be considered as a dimension reduction technique that tries to approximate the same topology of higher dimensional data in a lower dimensional space. Both approaches reduce dimensions but because of their different objectives they have not been successfully integrated. Here we show that in practice and more specifically in a supervised setting like classification, we can link the two methods to make random projection more informed in making the low dimensional representation competitive with the original data set with respect to classification accuracy. In this paper we propose a novel dimensionality reduction technique, namely informed weighted random projection, that combines Kernel PCA and random projection in an efficient way. The kernel PCA algorithm is applied initially to obtain a sub-space of reduced dimensions then the new lower dimensional bases derived by the kernel PCA are weighted in proportion to the measured robustness coefficient of each base. The proposed dimensionality reduction scheme has been applied on several benchmark datasets from the UCI repository and experimental results show that informed weighted random projection attains higher accuracy than the usual unweighted combination for all the datasets used in our experiments.