Topology-learnable graph convolution for skeleton-based action recognition

Highlights

• An effective method is proposed to generalize convolutions to the graph domain.

• The latent graph topologies beyond the handcrafted topology can be learned adaptively.

• A topology-learnable graph convolution network is constructed.

• The self-learning process makes the graph convolution more flexible and universal.

Abstract

Graph convolutional networks (GCNs) generalize convolutional neural networks into irregular graph-like structures. Generally, graph topologies are set by hand and fixed over all layers. Handcrafted connections may not be optimal and cannot fully use the self-learning ability of deep learning. In this work, we explore a topology-learnable graph convolution for skeleton-based action recognition. Specifically, a spatial graph convolution can be decomposed into a feature learning component that evolves the features of each graph vertex, and a graph vertex fusion component in which the latent graph topologies can be learned adaptively. Different initialization strategies for the learnable fusion matrix are evaluated. Experimental results that are based on the spatial-temporal GCNs for skeleton-based action recognition, demonstrate that convolution can work on graphs like on images, even if only a specific fusion matrix initialization that uses adjacency matrices is applied. Moreover, the self-learning process can learn the latent topology of a graph beyond the handcrafted topology, thereby making graph convolution flexible and universal.

Introduction

Convolutional neural networks (CNNs) [4] have achieved great success in machine learning fields, such as one-dimensional speech [5], two-dimensional images [11] and three-dimensional videos [27], where the underlying data representation has a grid-like structure. Local convolution filters with learnable parameters can be reused efficiently on all input positions. However, many fields involve data that cannot be represented in a grid-like structure. Such kinds of data generally lie in an irregular domain and can be usually represented in graph-based forms. Irregular structures prevent CNNs from being generalized into the graph domain.

Nevertheless, generalizing convolutions to the graph domain is an emerging topic in deep learning research. The advances in constructing graph convolutional networks (GCNs) on graphs are generally categorized into 1) spectral and 2) spatial perspectives. Spatial perspective defines convolutions directly on the graph within k-step neighbors [1], [2]. However, maintaining the weight sharing property of CNNs on different-sized neighborhoods is challenging. Generally, a transition matrix is constructed to define the neighborhood for each node, and a specific weight matrix is required for each node degree [1], [2]. Niepert et al. [18] assembled and normalized a fixed-size neighborhood for each node in the selected fixed-length sequence and learned neighborhood representations with CNNs. In such cases, GCNs are applied in the transductive setting with fixed graphs.

Handcrafted and normalized adjacency matrices are usually fixed over all layers when constructing GCNs. However, the handcrafted graph topology may not be optimal. Hamilton et al. [6] proposed an inductive framework that simultaneously learned the topological structure of each node’s neighborhood and the distribution of node features in the neighborhood. A set of aggregator functions is learned to aggregate the features from a node’s local neighborhood. Velickovic et al. [28] proposed an attention-based architecture for the node classification of graph-structured data. The proposed self-attention strategy learns the adaptive relationships between a node and its neighbors. This method makes the weight matrix of neighborhoods learnable. Shi et al. [23] overcame the constraint of local neighborhood by introducing the non-local idea into GCNs and constructed non-local GCNs based on spatial-temporal GCNs (ST-GCNs) [30] for skeleton-based action recognition.

Inspired by these works, we explore spatial graph convolutions based on the ST-GCN to fully use the self-learning ability of deep learning. Generally, spatial perspective graph convolutions can be implemented in two steps: (a) feature learning and (b) graph vertex fusion. The feature learning operations for each node can be implemented using a 1 × 1 2D-convolution [23], [30] or matrix multiplication [28]. Graph vertex fusion can be performed by matrix multiplication with a specifically initialized topology-learnable weight.

Unlike the non-local strategy [23], which breaks the constraint of local neighborhood by introducing the non-local idea into GCNs, we believe that the proposed topology-learnable graph convolution processes an inherent ability to break the local constraint. The non-local idea is redundant when the traditional k-step adjacency matrix is changed into topology-learnable ones. In contrast with the self-attention strategy [28], which learns the adaptive relationships between a node and its neighbors, we extend the adaptive relationships to the entire graph.

The remainder of this work is organized as follows. Section 2 reviews the related works. Section 3 provides a detailed description and analysis of the proposed topology-learnable graph convolution, followed by the experiments in Section 4. Finally, Section 5 presents the conclusion.