Elsevier

Information Sciences

Volume 560, June 2021, Pages 92-106
Information Sciences

Semi-supervised classification by graph p-Laplacian convolutional networks

https://doi.org/10.1016/j.ins.2021.01.075Get rights and content

Abstract

The graph convolutional networks (GCN) generalizes convolution neural networks into the graph with an arbitrary topology structure. Since the geodesic function in the null space of the graph Laplacian matrix is constant, graph Laplacian fails to preserve the local topology structure information between samples properly. GCN thus cannot learn better representative sample features by the convolution operation of the graph Laplacian based structure information and input sample information. To address this issue, this paper exploits the manifold structure information of data by the graph p-Laplacian matrix and proposes the graph p-Laplacian convolutional networks (GpLCN). As the graph p-Laplacian matrix is a generalization of the graph Laplacian matrix, GpLCN can extract more abundant sample features and improves the classification performance utilizing graph p-Laplacian to preserve the rich intrinsic data manifold structure information. Moreover, after simplifying and deducing the formula of the one-order spectral graph p-Laplacian convolution, we introduce a new layer-wise propagation rule based on the one-order approximation. Extensive experiment results on the Citeseer, Cora and Pubmed database demonstrate that our GpLCN outperforms GCN.

Introduction

With the advent of the information age and the growth of unstructured data, higher data dimensions and faster data updates are more prominent. How to extract effective and reasonable data information from massive datasets is an urgent problem to be solved. Therefore, the data information representation methods [1], especially the manifold structure information of data, has become an important research topic for machine learning. The goal of manifold learning [2] is to discover low-dimensional manifold structure from high-dimensional sampled data, i.e. it learns low-dimensional manifold in high-dimensional space, and then finds corresponding embedding mapping relationships to achieve data visualization. The related algorithms of manifold learning (ML) have an important research significance in theory and applications including machine learning [3], data mining [4], and computer vision [5].

In recent years, many prominent ML algorithms have been proposed and achieved great performance in the research hotspots of dimensionality reduction [6], clustering [7] and semi-supervised learning [8].

In the dimensionality reduction methods of ML, it maps the samples globally into a low-dimensional space by constructing the local neighborhood structure of the sample on the manifold. Tenenbaum et al. [9] utilized the geodesic distance in differential geometry to measure the pairwise distance between data. Roweis et al. [10] assumed that the low-dimensional manifold of the data sets and their mappings in the high dimensional observation space is locally linear. Then the dimensionality reduction is achieved by maintaining the fixed local linear relationships. Belkin et al. [11] described a manifold with an undirected weight graph, then it found a low-dimensional representation by using graph embedding.

The manifold clustering methods are mainly to find the potential manifold structure from the global or local correlation between data. The data will be clustered according to the data similarity for the manifold structure. Souvenir et al. [12] used the Isometric Feature Mapping (Isomap) method to calculate the distance between data, and solved the clustering problem through the expectation–maximization (EM) iterative method. In [13], the similarity of the data is measured by the distance how each sample moves to other samples on the manifold. Ye et al. [14] projected the data into a low-dimensional manifold that maximizes data separability, and then used the Mahalanobis distance as the distance measure in low-dimensional space.

The semi-supervised learning algorithms [15], [16] incorporate label information into the data manifold hypothesis, i.e. all samples are distributed on a low-dimensional manifold structure, and the geodesic distance for the sample with the same label is small. Liu et al. [17] used Hessian to preserve the manifold structure of data, which aimed to solve the poor extrapolating power of Laplacian. Cai et al. [18] took full advantage of the information of labeled and unlabeled samples to build a graph, which aimed to express a discrete approximation of the data manifold structure. The goal of this model is to learn a smooth discriminant function using the data manifold information. Kokiopoulou et al. [19] exploited the sparse geometry to express transformed manifold information of data.

Recently, deep learning has also received widespread concerns in the graph data field. Kipf et al. [20] proposed a generalization method of the convolutional neural network (CNN) on graph data, called graph convolutional networks (GCN). GCN can deal with arbitrary spatial structure data. It uses the graph Laplacian method to obtain the structure information of the graph data. GCN can extract representative data feature via automatically learning feature information and structure information of the graph data simultaneously. It is currently the best choice for the graph data learning task. Besides, the learning and computation complexity of the model has linear relationships. Different from GCN, two-order GCN [21] simultaneously utilized the direct and indirect neighbors relationships between nodes by using the two-order approximation of spectral graph Laplacian convolution. HyperGCN [22] generalized the simple graph convolution operation of GCN to the hypergraph domain by using hypergraph Laplacian to capture more complex or beyond pairwise relationships between nodes. Compared with the first-order derivatives of Laplacian, HesGCN [23] can acquire more accurate data features by the existence of the Hessian matrix’s two-order derivatives to get richer structure information in the original data with the complex structures. Graph attention network (GAT) [24] encoded the hidden representations of each node in the graph by introducing the self-attention mechanism to attend over its neighbors. Liu et al. [25] proposed a novel weakly supervised multi-label image classification framework based on GCN with learning the semantic label co-occurrence in an image.

However, the null space of the graph Laplacian is not rich. Thus, the extracted manifold structure relationship fails to have the representative capability of the graph data. Due to the lack of rich structure information, GCN cannot learn sufficient features from input samples (The learning process of GCN can be regarded as the fusion process of the input sample information and the structure information based on graph Laplacian). This reduces the classification performance of GCN as well. To better show the difference of graph Laplacian, graph p-Laplacian (p = 2) and graph p-Laplacian (p2) in preserving local geometry of data, Fig. 1 plots the experiment results of graph Laplacian, graph p-Laplacian (p = 2) and graph p-Laplacian (p2) based semi-supervised regression methods. The graph Laplacian has a bias towards the constant function and the extrapolation function remains unchanged along the spiral for unseen data, which cannot fit the data properly. When parameter p=2, graph p-Laplacian has a similar performance with graph Laplacian. Due to the variation smoothness of graph p-Laplacian’s (p2) extrapolation function with the geodesic distance, it can fit the data properly and extrapolates perfectly to unseen data.

In this paper, we exploit the graph p-Laplacian method to express the manifold structure of the data samples. The graph p-Laplacian [26], [27] is a nonlinear generalization of the graph Laplacian. It provides the most compelling theoretical evidence to better express the local structure information. In other words, graph p-Laplacian obtains tighter isoperimetric inequality, thus the upper and lower bounds on the second eigenvalue approximate the optimal Cheeger cut value well [28]. Compared with graph Laplacian, graph p-Laplacian can fit the data properly and extrapolates smoothly to unseen data with the geodesic distance, i.e. graph p-Laplacian has great superiority in the local manifold structure of data preserving. Besides, we apply the graph p-Laplacian matrix to spectral convolutions in the graph field and obtain a novel form of the spectral graph Laplacian convolutions, i.e. spectral graph p-Laplacian convolutions. Finally, we introduce a layer-wise rule representation form by optimizing the one-order approximation of the spectral graph p-Laplacian convolutions. Specially, we build a model by this layer-wise rule formula, i.e. graph p-Laplacian convolutional networks (GpLCN). It can learn rich sample features by integrating graph p-Laplacian into feature information. We test our model on Citation network datasets for node classification. Extensive experimental results prove that GpLCN obtains a higher recognition accuracy in comparison to GCN.

The main contributions of this paper can be summarized as follows:

  • 1)

    We generalize the spectral graph Laplacian convolutions into the spectral convolutions on graph p-Laplacian. It can utilize the graph p-Laplacian to better preserve the manifold structure information of data.

  • 2)

    Simplifying and deducing the one-order approximation of the spectral graph p-Laplacian convolutions, we propose an effective convolution layer rule.

  • 3)

    Based on the proposed convolution layer rule, we design a two-layer graph p-Laplacian convolutional networks (GpLCN).

  • 4)

    Extensive experiment results on public datasets show that the proposed GpLCN outperforms several semi-supervised classification methods such as GCN, HyperGCN.

The rest of this paper is organized as follows. Section 2 briefly describes related works on graph p-Laplacian theory and graph convolutional network. Section 3 presents the definition of spectral graph p-Laplacian convolutions and GpLCN in detail. Experimental results and analysis of GpLCN are shown in Section 4. Finally, Section 5 concludes this paper.

Section snippets

Related work

Our GpLCN is inspired by graph p-Laplacian theory and GCN [20]. We give a brief description of the related works in this section.

Graph p-Laplacian convolutional networks

Inspired by the definition of spectral convolution on graph data and the GCN model, we introduce a novel definition on the graph, i.e. spectral graph p-Laplacian convolution. Moreover, we introduce a new layer-wise propagation rule form via the one-order approximation of spectral graph p-Laplacian convolutions. Specifically, we build a new model by using the multiple layer-wise propagation formula, named graph p-Laplacian convolutional networks (GpLCN).

The GpLCN algorithm can be divided into

Experiments

In this section, to demonstrate the performance of the proposed GpLCN model, we conduct substantial experiments for node classification on citation network databases, such as Citeseer, Cora and Pubmed [37]. The description of the citation network datasets is shown in detail. The parameter settings of the GpLCN model are then given. We finally provide the experiment results.

Conclusion

We have proposed a p-Laplacian method to preserve the local structure information of data. The null space of p-Laplacian is rich, and it can use the geodesic distance to extrapolate smoothly for unseen data. We applied the graph p-Laplacian matrix to spectral graph convolution, i.e. spectral graph p-Laplacian convolution. Finally, we have proposed a new model graph p-Laplacian convolutional networks (GpLCN) for node classification. The GpLCN is based on the one-order approximation for spectral

CRediT authorship contribution statement

Sichao Fu: Conceptualization, Methodology, Formal analysis, Writing - original draft. Weifeng Liu: Supervision, Funding acquisition, Writing - review & editing. Kai Zhang: Supervision, Funding acquisition, Supervision. Yicong Zhou: Supervision, Funding acquisition, Writing - review & editing, Writing - review & editing. Dapeng Tao: Supervision, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61671480, in part by the Major Scientific and Technological Projects of CNPC under Grant ZD2019-183-008, in part by the Yunnan Natural Science Funds under Grant 2018FY001(-013) and Grant 2018YDJQ004, and by the Open Project Program of the National Laboratory of Pattern Recognition (NLPR) (Grant No. 202000009).

References (49)

  • J.B. Tenenbaum et al.

    A global geometric framework for nonlinear dimensionality reduction

    Science

    (2000)
  • S.T. Roweis et al.

    Nonlinear dimensionality reduction by locally linear embedding

    Science

    (2000)
  • M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques for embedding and clustering, in: Proc. Adv. Neural...
  • R. Souvenir, R. Pless, Manifold clustering, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), volume 1, 2005, pp....
  • M. Breitenbach, G. Z. Grudic, Clustering through ranking on manifolds, in: Proc. Int. Conf. Mach. Learn. (ICML), 2005,...
  • J. Ye, Z. Zhao, H. Liu, Adaptive distance metric learning for clustering, in: Proc. IEEE Conf. Comput. Vis. Pattern...
  • J.E. Van Engelen et al.

    A survey on semi-supervised learning

    Mach. Learn.

    (2020)
  • W. Liu et al.

    Human activity recognition by manifold regularization based dynamic graph convolutional networks

    Neurocomputing

    (2020)
  • W. Liu et al.

    Multiview hessian regularization for image annotation

    IEEE Trans. Image Process.

    (2013)
  • D. Cai, X. He, J. Han, Semi-supervised discriminant analysis, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2007, pp....
  • E. Kokiopoulou et al.

    Optimal image alignment with random projections of manifolds: algorithm and geometric analysis

    IEEE Trans. Image Process.

    (2011)
  • T.N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: Proc. IEEE Int. Conf....
  • S. Fu et al.

    Two-order graph convolutional networks for semi-supervised classification

    IET Image Process.

    (2019)
  • N. Yadati, M. Nimishakavi, P. Yadav, A. Louis, P. Talukdar, Hypergcn: Hypergraph convolutional networks for...
  • Cited by (43)

    View all citing articles on Scopus
    View full text