Elsevier

Applied Soft Computing

Volume 16, March 2014, Pages 112-123
Applied Soft Computing

Perceptual relativity-based semi-supervised dimensionality reduction algorithm

https://doi.org/10.1016/j.asoc.2013.12.004Get rights and content

Highlights

  • The perceptual relativity has been applied to improve the performance of classification on the sparse, noisy or imbalanced data, indicating the possibility of other perceptual laws in cognitive psychology being considered for classification.

  • A novel dimensionality reduction method has been designed for semi-supervised dimensionality reduction combined with relative transformation. It is more adaptive to parameters selection problem and obtain better performance.

Abstract

As we all know, a well-designed graph tends to result in good performance for graph-based semi-supervised learning. Although most graph-based semi-supervised dimensionality reduction approaches perform very well on clean data sets, they usually cannot construct a faithful graph which plays an important role in getting a good performance, when performing on the high dimensional, sparse or noisy data. So this will generally lead to a dramatic performance degradation. To deal with these issues, this paper proposes a feasible strategy called relative semi-supervised dimensionality reduction (RSSDR) by utilizing the perceptual relativity to semi-supervised dimensionality reduction. In RSSDR, firstly, relative transformation will be performed over the training samples to build the relative space. It should be indicated that relative transformation improves the distinguishing ability among data points and diminishes the impact of noise on semi-supervised dimensionality reduction. Secondly, the edge weights of neighborhood graph will be determined through minimizing the local reconstruction error in the relative space such that it can preserve the global geometric structure as well as the local one of the data. Extensive experiments on face, UCI, gene expression, artificial and noisy data sets have been provided to validate the feasibility and effectiveness of the proposed algorithm with the promising results both in classification accuracy and robustness.

Introduction

Real world data, such as digital photographs, gene expression profile, face data sets and web text, usually have the character of high dimensionality. To avoid the problem of “curse of dimensionality” possibly existing in high-dimensional tasks, dimensionality reduction is often conducted [1]. Dimensionality reduction is widely recognized as one of the key steps in areas such as computer vision, machine learning and pattern recognition and its main goal is to map high dimensional data into a meaningful representation of lower dimensional space in which important features are preserved. Various useful feature extraction methods have been proposed for decades, among which Principal Component Analysis (PCA) [2] and Linear Discriminate Analysis (LDA) [3] are two most classical and well-known linear subspace learning methods. As an unsupervised method, PCA seeks projections with the covariance of samples maximally preserved and meanwhile ensures the extracted features have least reconstruction error. In contrast to PCA, LDA is a supervised method, which aims to search for a set of projection vectors such that the ratio of between-class scatter to within-class scatter is maximized. For the use of class label information, LDA is often more powerful in discriminant applications than PCA. However, if instances sampled from a space confined to nonlinear subspace, the embedding results of these two methods are both distorted. So with the presentation and development of manifold learning, lots of nonlinear feature extraction methods such as LLE [4], ISOMAP [5], LTSA [6], and so on, have been proposed, to address this problem.

It is notable that in many practical classification tasks, labeled samples are fairly expensive to obtain as labeling often requires expensive human labor and much time, and meanwhile unlabeled samples are very easy to collect. However, in supervised learning, the unlabeled samples are completely excluded from the training process, which leads to a potential waste of valuable classification information buried in unlabeled samples. Therefore, “Semi-supervised Learning” with both labeled and unlabeled data has recently attracted more and more attention [7]. The goal of semi-supervised classification is to use unlabeled data to improve the generalization of the classifier. By using unlabeled samples, semi-supervised approaches usually have better generalization ability than the corresponding supervised ones. And, due to the extra focus on labeled samples, semi-supervised approaches often possess higher performance than the unsupervised ones. As we all know, pairwise constraints information also called side-information is more general than label information, since we can obtain side information from label information but not vice verse [8]. So learning with side information is becoming an important area in the machine learning. With this in mind, Zhang et al. [9] proposed semi-supervised dimensionality reduction (SSDR), which utilizes both the must-link and cannot-link constraints effectively. However, it fails to preserve the local structure of the data except preserving the global covariance structure. Even worse, SSDR is sensitive to noise and outliers, because it considers the similarity between samples coarsely. Cevikalp et al. [10] proposed constrained locality preserving projections (CLPP) recently, which can use the must-link and cannot-link information and also use the unlabelled data by preserving local structure. However, there's no reliable approach to determine an optimal parameter t to construct the adjacency graph when there exist noise and outliers of samples. Wei and Peng [11] proposed a method named the neighborhood preserving-based semi-supervised dimensionality reduction algorithm (NPSSDR), which makes full use of side information, not only preserves the must-link and cannot-link constraints but also can preserve the local structure of the input data in the low dimensional embedding subspace. However, it is still susceptible to noise and outliers as it equally treats the noisy samples and normal samples. All of these methods mentioned above are variants of graph embedding [12]. Graph construction is crucial to graph embedding, so constructing a faithful graph of samples for graph embedding is promising. Unfortunately, as the dimensionality of sample increases, the distance metric of samples becomes meaningless. Consequently, the graph structure of samples is deteriorated.

Meanwhile even worse, in real-world life, there will be abundant of the sparse, noisy and imbalance data, which will influence the construction of graph and hence lead to the degraded performance of semi-supervised dimensionality reduction. Considering that past semi-supervised dimensionality reduction methods are sensitive to the selection of neighborhood parameter and rely more on the construction of graph, a novel algorithm of relative semi-supervised dimensionality reduction (RSSDR) is proposed from a different respective in this paper, to overcome this problem by utilizing the perceptual relativity in terms of cognitive psychology to semi-supervised dimensionality reduction. It performs the relative transformation [13] over the training samples to build the relative space. Subsequently the algorithm set the edge weights of neighborhood graph through minimizing the local reconstruction error in the relative space and can preserve the global geometric structure of the sampled data set as well as preserving its local one. The feasibility and effectiveness of RSSDR are verified on the high-dimensional, sparse and noisy data sets with the promising results in classification accuracy and robustness.

There are two main contributions of RSSDR in this paper: (1) the perceptual relativity has been applied to improve the performance of classification on the sparse, noisy or imbalance data, indicating the possibility of other perceptual laws in cognitive psychology being considered for classification. (2) A novel dimensionality reduction method has been designed for semi-supervised dimensionality reduction combined with relative transformation. It is more adaptive to parameters selection problem and obtains better performance. Therefore, RSSDR is robust to noise and less sensitive to choice of parameters. So it is more applicable to real world tasks.

The rest of this paper is organized as follows: Section 2 presents some basic concepts. A novel method is designed in Section 3. The proposed method is evaluated through experiments in Section 4. The paper is concluded with a summary and discussion of possible future work in Section 5.

Section snippets

Elementary concepts

In this section, we present two concepts that our approach is based on and which will serve as building blocks.

The problem setting

Here, the side-information based semi-supervised linear dimensionality reduction problem is defined as follows. First, supposing there is a set of D-dimension points X = {x1, x2, …, xn} T, xi  RD, together with some pairwise must-link (M) and cannot-link (C) constraints as side-information: M : (xi, xj)  M, if xi and xj both belong to the same class; and, C : (xi, xj)  C, if xi and xj both belong to different classes. Then, a transformation matrix W (W=w1,w2,...,wdRD×d(dD)) is found such that the

Experimental setup

In this section, to validate the proposed RSSDR, we conduct several experiments to compare the effectiveness of RSSDR with other dimensionality reduction algorithms: NPSSDR, SSDR, CLPP, PCA, the first three methods are semi-supervised with pairwise constraints, and the leftover is unsupervised. The comparative index is classification accuracy of the nearest neighborhood classifier on the testing samples after dimensionality reduction techniques are applied. The Baseline method is simply

Conclusions and future work

Graph's construction plays an important role in graph-based semi-supervised learning methods. Fortunately, graph's construction has received substantial attention especially in recent years. Inspired by the good performance of relative transformation method on classification, this paper defines a novel method RSSDR integrating both relative transformation and semi-supervised dimensionality reduction, which considers the graph construction among data samples in the relative space instead of

Acknowledgements

The research leading to these results has received the support of the National Natural Science Foundation of China under grants no. 61070090, 61273363, 61003174, 60973083, a grant from the Guangdong Natural Science Funds for Distinguished Young Scholar (No. S2013050014677), a grant from the Fundamental Research Funds for the Central Universities (2014G0007), a grant from China Postdoctoral Science Foundation (No. 2013M540655), a grant from NSFC-Guangdong Joint Fund (No. U1035004), and two

Xianfa Cai spent his undergraduate days at Jiangxi Normal University from 1998 to 2002, went on to get his master's degree from SunYat-sen University from 2003 to 2006, and a Ph.D. candidate at the School of Computer Science and Engineering, South China University of Technology. He is now a lecturer at the School of Medical Information Engineering, Guangdong Pharmaceutical University. His current research interests include machine learning, pattern recognition, bioinformatics.

References (26)

  • G. Wen et al.

    Relative transformation-based neighborhood optimization for isometric embedding

    Neurocomputing

    (2009)
  • G. Wen et al.

    Cognitive gravitation model for classification on small noisy data

    Neurocomputing

    (2013)
  • R.O. Duda et al.

    Pattern Classification

    (2001)
  • I.T. Jolliffe

    Principal component analysis

    Springer Series in Statistics

    (2002)
  • A.M. Martinez et al.

    PCA versus LDA

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2001)
  • S. Roweis et al.

    Nonlinear dimensionality reduction by locally linear embedding

    Science

    (2000)
  • J.B. Tenenbaum et al.

    A global geometric framework for nonlinear dimensionality reduction

    Science

    (2000)
  • Z. Zhang et al.

    Principal manifolds and nonlinear dimensionality reduction via tangent space alignment

    SIAM Journal of Scientific Computing

    (2004)
  • X. Zhu, Semi-Supervised Learning Literature Survey [EB/OL], Available from...
  • D. Klein et al.

    From instance-level constraints to space-level constraints: making the most of prior knowledge in data clustering

  • D.Q. Zhang et al.

    Semi-supervised dimensionality reduction

  • H. Cevikalp et al.

    Semi-supervised dimensionality reduction using pairwise equivalence constraints

  • J. Wei et al.

    Neighbourhood preserving based semi-supervised dimensionality reduction

    Electronics Letters

    (2008)
  • Cited by (4)

    • Augmenting features by relative transformation for small data

      2021, Knowledge-Based Systems
      Citation Excerpt :

      Another application of RT is the semi-supervised dimensionality reduction. It is firstly applied to build the relative space on the training data and then construct the neighborhood graph through minimizing the local reconstruction error in the relative space [37]. Our another work generates features by calculating distances between the test sample and its neighborhoods in each class as features and then these features are combined with the original features to form the new feature vector for the sample [50].

    • Semi-supervised rough fuzzy Laplacian Eigenmaps for dimensionality reduction

      2019, International Journal of Machine Learning and Cybernetics

    Xianfa Cai spent his undergraduate days at Jiangxi Normal University from 1998 to 2002, went on to get his master's degree from SunYat-sen University from 2003 to 2006, and a Ph.D. candidate at the School of Computer Science and Engineering, South China University of Technology. He is now a lecturer at the School of Medical Information Engineering, Guangdong Pharmaceutical University. His current research interests include machine learning, pattern recognition, bioinformatics.

    Guihua Wen, born in 1968, is a Ph.D., professor and doctor supervisor. In 2005–2006, he did visiting research on machine learning and semantic web in School of Electronics and Computer, University of Southampton, UK. His main research interests are computational creativity, data mining and knowledge discovery, machine learning, and cognitive geometry. Since 2006, he proposed some original methods based on the computation of cognitive laws, which can effectively solve difficult problems in information science. The research results have been published in the international journals, including Pattern Recognition, Neurocomputing, Journal of Software, Journal of computer Research and Development. He also published some papers in the international conferences such as IJCAI. Since 2006, he directed the projects from the China National Natural Science Foundation, State Key Laboratory of Brain and Cognitive Science, the Ministry of Education Scientific Research Foundation for returned overseas students, Guangdong Provincial Science and Technology research project, the Fundamental Research Funds for the Central Universities, SCUT. He also directed many projects from enterprises, with applications of his research results to the practical problems. He has ever been a Council Member of Chinese Association for Artificial Intelligence and a program committee member of many international conferences. He is also a reviewer for China National Natural Science Foundation.

    Jia Wei received his B.Sc. and M.Sc. degrees both in Computer Science from Harbin Institute of Technology in 2003, 2006 respectively, and his Ph.D. degree in Computer Science from South China University of Technology in 2009. He is now a lecturer at the School of Computer Science and Engineering, South China University of Technology. His research fields include machine learning and images retrieval.

    Jie Li received her B.Sc. in Mechanical Engineer and Automation in 1999, M.Sc. in Computer-Aided Material Machining in 2002, and Ph.D. degrees in Mechanical Designing and Theory in 2006, all from South China University of Technology, Guangzhou, China. Her current research interests include machine learning, pattern recognition and numerical simulation; she has published over 10 research articles.

    Zhiwen Yu is a professor in the School of Computer Science and Engineering, South China University of Technology, Guangzhou, China. He received the B.Sc. and M.Phil. degrees from the SunYat-sen University in China in 2001 and 2004 respectively, and the Ph.D. degree in Computer Science from the City University of Hong Kong, in 2008. His research interests include pattern recognition, bioinformatics, multimedia, intelligent computing and data mining. He has published more than 50 technical articles in referred journals and conference proceedings in the areas of pattern recognition, bioinformatics and multimedia.

    View full text