Dimensionality reduction via preserving local information

https://doi.org/10.1016/j.future.2018.01.016Get rights and content

Highlights

  • A local similarity preserving (LSP) function for finding appropriate neighbors is proposed.

  • The proposed algorithms were able to preserve the similarity information and geometric structure of the samples.

  • Effectiveness and efficiency was verified upon extensive experiments using digits and face datasets.

Abstract

Dimensionality reduction is an important technique for multimedia data in smart cities. As a significant type of multimedia data, images are being processed frequently using such technique. It is worth noting that finding neighbors is an integral part of graph embedding algorithms in dimensionality reduction. In this paper, we present an effective and efficient metric function, named local similarity preserving (LSP), which can preserve the similarity information of each sample to its homogeneous and heterogeneous neighbors. The proposed LSP function is helpful to enhance the discriminative capability of subsequent feature extractors. Based on LSP function, we also propose two novel algorithms, local similarity preserving discriminant (LSPD) algorithm, which can preserve the local similarity information and LSPD+ algorithm, which can preserve the local similarity and geometric structure information, respectively. The experimental results on digit visualization and face recognition demonstrate the effectiveness and efficiency of the proposed algorithms.

Introduction

In many applications, many multimedia data in smart cities are characterized by hundreds or even thousands of features, such as images. However, directly using of high dimensional data may significantly degrade the performance of many applications. Dimensionality reduction, as a crucial technique, has yielded a surge of interest research. It is worth noting that finding neighbors is an essential step of graph embedding algorithms for dimensionality reduction. Since Roweis et al. [1] find k nearest neighbors to reconstruct each sample xi to preserve the local geometric structure, which could seek a low-dimensional embedding of high-dimensional inputs, there are many proposed graph embedding algorithms for dimensionality reduction, which reconstruct the samples by its neighbors or preserve the local structure by finding nearest neighbors, such as [[2], [3], [4], [5], [6], [7]]. However, they are all unsupervised algorithms, which cannot work well in the task of classification. To tackle the unsupervised problem, many algorithms have been proposed [[8], [9], [10], [11], [12], [13]], such as [13] proposed a new multi-view discriminative manifold embedding method. They all can preserve geometric structure of samples. However, they ignore the similarity information, which is important in the task of visualization and classification.

Fig. 1 gives a vivid diagram to illustrate the procedure of graph embedding. In Fig. 1, there are two ways finding appropriate neighbors to construct adjacency graphs: Fig. 1 (a1), which is the traditional way constructing adjacency graph; and Fig. 1 (b1), which is the ideal way constructing adjacency graph. And we also have two results: Fig. 1 (a2), which is the result of constructing the adjacency graph with the traditional way; and Fig. 1 (b2), which is the result of constructing the adjacency graph with the ideal way. Our purpose is to make the representations of these two samples as close as possible in the new space whether they are close or not in the high-dimensional space. However, traditional way finds nearest neighbors, measured by Euclidean, constructing adjacency graph. Thus, if two same class samples have a far distance, they cannot establish a relationship, so that they will not be close in the subspace. In the task of classification, we should give priority to the distant samples in the same class, which will make the classification ability of the algorithm seriously deteriorate if we ignore them. The fundamental challenge is how to find the appropriate neighbors to establish the relationship to achieve the same result of ideal way.

Inspired by recent advances [[14], [15], [16], [17], [18], [19], [20], [21]], in this paper, we propose a local similarity preserving (LSP) function for finding appropriate neighbors, which can preserve the similarity information of the samples. Based on LSP function, we also propose two algorithms, LSPD algorithm, which finds the appropriate neighbors and preserves the similarity information by using LSP function; and LSPD+ algorithm, which also preserves the geometric structure of data based on LSPD. Extensive experiments demonstrate that the effectiveness and efficiency of our algorithms comparing with the state-of-the-art algorithms.

The main contribution of our work is that we propose a local similarity preserving function and two algorithms, LSPD and LSPD+ based on LSP function. Both LSPD and LSPD+ outperform the related state-of-the-art algorithms in digits and face data sets. Moreover, we analyze the experiment results, which demonstrate the effectiveness and efficiency of our proposed algorithms.

Section snippets

Related work

Our work is inspired by graph embedding for dimensionality reduction. Therefore, we briefly review some classical algorithms, which are closely related to our work.

Many graph embedding algorithms are proposed in recent years, such as discriminant neighborhood embedding [22], exponential local discriminant embedding [23], orthogonal neighborhood preserving projections [24], locality sensitive discriminant analysis [25], marginal fisher analysis [[26], [27]], local feature discriminant

LSP for graph embedding algorithms

In this section, a local similarity preserving (LSP) function for finding appropriate neighbors is proposed to preserve the local similarity information, and we also present the details of the proposed algorithms.

Experiments

In this section, several experiments are conducted to validate the effectiveness and efficiency of the proposed algorithms.

Conclusion

In this paper, we present a novel local similarity preserving function for preserving the local similarity information in the process of dimensionality reduction. On the basis of LSP function, we further propose two algorithms, LSPD, which preserves the local similarity information of data, and LSPD+, which not only preserves the locality similarity information of samples, but also keeps the geometric structure, respectively. Extensive experiments on visualization and classification show the

Acknowledgments

This work is supported in part by the National Science Foundation of China (Grant No. 61472047).

Shangguang Wang received his Ph.D. degree at Beijing University of Posts and Telecommunications in 2011. He is an associate professor at the State Key Laboratory of Networking and Switching Technology (BUPT). He has published more than 100 papers, and played a key role at many international conferences, such as general chair and PC chair. His research interests include service computing, cloud computing, and mobile edge computing. He is a senior member of the IEEE, and the Editor-in-Chief of

References (35)

  • X.F. He, D. Cai, S.C. Yan, H.J. Zhang, Neighborhood-preserving embedding, in: Proc. the 10th IEEE International...
  • YangJ. et al.

    Globally maximizing, locally minimizing: Unsupervised discriminant projection with applications to face and palm biometrics

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2007)
  • H.T. Chen, H. Wei, T.L. Liu, Local discriminant embedding and its variants, in: Proc. the IEEE Computer Society...
  • F.P. Nie, S.M. Xiang, C.S. Zhang, Neighborhood minMax projection, in: Proc. the 20th international Joint Conference on...
  • XiangS.M. et al.

    Nonlinear dimensionality reduction with local spline embedding

    IEEE Trans. Knowl. Data Eng.

    (2009)
  • NikitidisS. et al.

    Maximum margin projection subspace learning for visual data analysis

    IEEE Trans. Image Process.

    (2014)
  • WangX.Z. et al.

    Multi-view discriminative manifold embedding for pattern classification

    J. Intell. Comput.

    (2017)
  • Cited by (14)

    • Supervised discriminative dimensionality reduction by learning multiple transformation operators

      2021, Expert Systems with Applications
      Citation Excerpt :

      Locally Preserving Projection (LPP) is a popular example of graph embedding methods, preserving the local structure of data in the reduced subspace (He & Niyogi, 2004). While the original version of LPP is unsupervised, there are several supervised extensions of LPP (Wang, Ding, Hsu, & Yang, 2020; Zhang, Xue, Lu, & Guo, 2006). Double graphs-based discriminant projection (DGDP) is a recent similar work, which constructs two adjacent graphs instead of one.

    • Fault diagnosis of rotor based on Local-Global Balanced Orthogonal Discriminant Projection

      2021, Measurement: Journal of the International Measurement Confederation
      Citation Excerpt :

      In the light of Fig. 3, the regulate parameters (α and β) are set to be α = 0.9 and β = 0.9. Visual comparison of low-dimensional features: In order to verify the dimensionality reduction and feature extraction effect of LGBODP, LPP, SOLFDA [33], MNMP [34], LSPD [35], ODP and LGBODP algorithms are used to perform dimensionality reduction comparison experiments. In this experiment, the parameter values of each method are obtained by combining grid search method and cross-validation method.

    • Discriminative globality and locality preserving graph embedding for dimensionality reduction

      2020, Expert Systems with Applications
      Citation Excerpt :

      LPP is a linear extension of Laplacian eigenmap (LE) that is manifold learning (Belkin & Niyogi, 2003). Since LPP can well reflect the local geometry of high-dimensional data, its many variants were introduced, such as in Yang, Zhang, Yang, and Niu (2007), Zhang, Xue, Lu, and Guo (2006), Xu, Zhong, Yang, and Zhang (2010), Gao, Liu, Zhang, Gao, and Li (2013), Gou and Zhang (2013), Wang et al. (2017b), Ding and Zhang (2015), Wang, Ding, Hsu, and Yang (2018), Lu, Ding, Xu, and Wang (2018), Yang, Liu, Yu, and Wang (2018) and Wang et al. (2017a). Yang et al. proposed unsupervised discriminant projections as a simplified version of LPP (Yang et al., 2007).

    View all citing articles on Scopus

    Shangguang Wang received his Ph.D. degree at Beijing University of Posts and Telecommunications in 2011. He is an associate professor at the State Key Laboratory of Networking and Switching Technology (BUPT). He has published more than 100 papers, and played a key role at many international conferences, such as general chair and PC chair. His research interests include service computing, cloud computing, and mobile edge computing. He is a senior member of the IEEE, and the Editor-in-Chief of the International Journal of Web Science.

    Chuntao Ding received the B.S. and M.S. degrees from SIAS international University in 2012 and Soochow University in 2015, respectively, both in software engineering. He is currently a Ph.D. at the State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications (BUPT). His research interests include machine learning, mobile edge computing.

    Ching-Hsien Hsu is a professor in the Department of Information Engineering and Computer Science, Feng Chia University, Taichung, Taiwan; He was distinguished chair professor at Tianjin University of Technology, China, during 2012–2016.  His research includes high performance computing, cloud computing, parallel and distributed systems, big data analytics and intelligence. He has published 200 papers in these areas, including top journals such as IEEE TPDS, IEEE TSC, IEEE TCC, IEEE TETC, IEEE T-SUSC, IEEE Systems, IEEE Network, IEEE Communications, ACM TOMM.  Dr. Hsu is serving as editorial board for a number of prestigious journals, including IEEE TSC, IEEE TCC, IJCS, JoCS.  He has been acting as an author/co-author or an editor/co-editor of 10 books from Elsevier, Springer, IGI Global, World Scientific and McGraw-Hill.  Dr. Hsu was awarded nine times distinguished award for excellence in research from Chung Hua University.  He is vice chair of IEEE TCCLD, executive committee of IEEE TCSC, Taiwan Association of Cloud Computing and an IEEE senior member.

    Fangchun Yang received his Ph.D.in communications and electronic systems from the Beijing University of Posts and Telecommunication in 1990. He is currently professor at the Beijing University of Posts and Telecommunication, China. He has published six books and more than 80 papers. His current research interests include network intelligence, service computing, communications software, soft-switching technology, and network security. He is a Fellow of the IET.

    View full text