GLViG: Global and Local Vision GNN May Be What You Need for Vision

Abstract

In this article, we propose a novel vision architecture termed GLViG, which leverages graph neural networks (GNNs) to capture local and important global information in images. To achieve this, GLViG represents image patches as graph nodes and constructs two types of graphs to encode the information, which are subsequently processed by GNNs to enable efficient information exchange between image patches, resulting in superior performance. In order to address the quadratic computational complexity challenges posed by high-resolution images, GLViG adaptively samples the image patches and optimizes computational complexity to linear. Finally, to enhance the adaptation of GNNs to the 2D image structure, we use Depth-wise Convolution dynamically generated positional encoding as a solution to the fixed-size and static limitations of absolute position encoding in ViG. The extensive experiments on image classification, object detection, and image segmentation demonstrate the superiority of the proposed GLViG architecture. Specifically, the GLViG-B1 architecture achieves a significant improvement on ImageNet-1K when compared to the state-of-the-art GNN-based backbone ViG-Tiny (80.7% vs. 78.2%). Additionally, our proposed GLViG model surpasses popular computer vision models such as Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Vision MLPs. We believe that our method has great potential to advance the capabilities of computer vision and bring a new perspective to the design of new vision architectures.

A Appendix

1.1 A.1 Semantic Segmentation

Settings: We chose the ADE20K dataset [53] to evaluate the semantic segmentation performance of GLViG. The ADE20K dataset consists of 20,000 training images, 2,000 validation images, and 3,000 test images, covering 150 semantic categories. For our framework, we utilized MMSEG [5] as the implementation framework and Semantic FPN [18] as the segmentation head. All experiments were conducted using 4 NVIDIA 3090 GPUs. During the training phase, the backbone was initialized with weights pre-trained on ImageNet-1K, while the newly added layers were initialized with Xavier [8]. We optimized our model using AdamW [28] with an initial learning rate of 1e-4. Following common practices [2, 18], we trained our models for 40,000 iterations with a batch size of 32. The learning rate was decayed using a polynomial decay schedule with a power of 0.9. In the training phase, we randomly resized and cropped the images to a size of 512 $\times$ 512. During the testing phase, we rescaled the images to ensure the shorter side was 512 pixels.

Table 4. Results of semantic segmentation on ADE20K [53] validation set. We calculate FLOPs with input size 512 $\times$ 512 for Semantic FPN.

Results: As shown in Table 4, GLViG outperforms the representative backbones in Semantic Segmentation, including ResNet [12] (CNN), Attention-based method PVT [41] (Vision Transformer). Considering no relevant data is available, we didn’t compare ViG [9] to the proposed method. For example, GLViG-B1 outperforms ResNet-18 by 7.4% mIoU (40.3% vs. 32.9%) and PVT-Tiny by 4.6% mIoU (40.3% vs.35.7%) under comparable parameters and FLOPs. In general, GLViG performs competently in semantic segmentation and consistently outperforms various backbone models (Tables 5 and 6).

1.2 A.2 Details of the GLViG Architecture

Table 5. Detailed settings of GLViG series.

Table 6. Training hyper-parameters of GLViG for ImageNet-1K [31].