GGAC: Multi-relational image gated GCN with attention convolutional binary neural tree for identifying disease with chest X-rays

Highlights

• Use Graph Convolutional Network to learn high-dimensional feature.

• Mining the discriminative features and multi-modal relations of images.

• Solve the problem of weight distribution in different neighborhoods.

• Image relations generate more image representations for images.

Abstract

Using medical images for disease identification is an important application in the medical field. Graph Convolutional Network (GCN) is proposed to model multi-relational image and generate more informative image representations. Recently, the relations between medical images are utilized to identify diseases. This paper proposes a Gated GCN with Attention Convolutional Binary Neural Tree (GGAC) for Multi-Relational Image Identifying Disease. GGAC extracts the discriminative features of the image, strengthen the ability to model medical images, understands images representation deeply and then well captures the multi-modal relation between images. Firstly, an Attention Convolutional Binary Neural Tree based on the attention mechanism is designed to extract the fine-grained features of the images, and use the attention conversion operation on the edge of the tree structure to enhance the network’s acquisition of key image features. Secondly, a Gated GCN is proposed to improve GCN performance by solving the problem of the weight distribution of different neighbors in the same-order neighborhood. Thirdly, a GCN propagation rule is used to transfer messages in multi-relational Graph and then solves the message passing problem of high-dimensional feature data in GCN. Finally, we verify GGAC on a multi-relational graph constructed on the Chest X-rays14. It can be seen from the experiment that overfitting and underfitting can be solved to a certain extent through the extraction and inference of the features of the multi-relational graph, and then GGAC has better performance than the state-of-the-art methods, and keeps good in model complexity.

Introduction

The basic tasks of computer vision include image classification, object localization, semantic segmentation, and instance segmentation. Convolutional neural networks and annotated image datasets have made significant progress in learning image representation classification in [1], [2], [3]. However, most of the existing deep learning methods neglect the relationships among images, which can benefit extraction of helpful semantic information from associations among images in reality. Meanwhile, there are one or more potential relationships among chest X-rays which can be regarded as a piece of auxiliary information that is helpful for further judging and analyzing chest X-rays. In clinical environments, diagnosis can be easily made by exploring such intrinsic relationships among chest X-rays.

We modeled Chest X-rays images and the relationships between them as a Multi-Relational Chest X-rays Graph where each node corresponds to a Chest X-ray image and the edges between two nodes represent the multiple relationships. The edges between two Chest X-ray images describe the close relations between the images. The construction of Multi-Relational Chest X-rays Graph and the definition of the relationships are given in Section 4.2. Figure 1(a) shows a Chest X-rays Graph with 5 nodes and 3 types of relationships. Our GGCN in Fig. 1(b) processes this multi-relational graph by extracting the features of the images and obtain the semantic relationships between the images. This process provides strong support for the segmentation, classification and object detection of images shown in Fig. 1(d). It can be seen from the experiment in Section 4 that over fitting and under fitting can be solved to a certain extent through the extraction and inference of the features of the multi-relational graph.

There are multiple relationships between the nodes in a Multi-relational Chest X-rays Graph. For example, there are plenty of edges between the two nodes in the Graph if the patients corresponding to image 1 and image 2 in Chest X-rays Graph have the same age and gender. We use the multiple relationships between Multi-relational Chest X-rays Graph to make the diagnosis of diseases more accurate. At present, some emerging researches on graph convolution [4], [5], [6], [7], [8], [9] modeled graph data which use the characteristics and the structural information of nodes to represent nodes. Although the application of GCN [4] to Chest X-ray Graphs is extremely challenging, many scholars have made attempts and achieve success in this regard. Graph SAGE [5] solved the problem of GCN in inductive learning on large images; Relational GCN [7] solved the multi-relational problem where each edge has a label and direction associated with it. Inspired by the ideas of Graph SAGE and relational GCN, the features of the original images were applied and the relationships between images was used to update the information in the Chest X-rays Graph. In the original GCN, the messages passing between adjacent nodes is obtained through linear transformation, but the nodes in Chest X-rays Graph are graphs with high-dimensional features, so an Encoder was used to transfer in multi-relational Graph. In Encoder, the use of the Attention Convolutional Binary Neural Tree was proposed to extract the fine-grained features of the images, then the attention conversion operation on the edge of the tree structure to strengthen the ability of network to acquire key features of images was used. We further design a Gated GCN Block based on GCN to apply to the GGAC. When modeling the relationships between nodes, Gated GCN solves the problem of weight distribution of different neighbors in the same-order neighborhood, and strengthens the generalization ability of the overall architecture in the graph structure. The addition of Gated GCN Block makes the model focus more on neighbor relations that are beneficial to the results. Finally, we select the Chest X-ray8 data set [10] to verify GGAC due to the rich relationships between these images.

This paper introduces the gating mechanism into GCN to solve the problem of weight distribution in same-order neighborhoods while Dauphin et al. [11] introduced the gating mechanism into CNNs to reduced the gradient dispersion and preserved the ability of nonlinearity. And the gating mechanism into GCN can enhances the ability of the model to capture spatial information relevance, and broadens the application of the gating mechanism in GCN. We designed an Attention Convolutional Binary Neural Tree into GCN to capture the discriminative features in medical images, while Wang et al. [12] focused on the interactions that occurred by introducing two additional sources of information, namely the spatial location of the candidate objects and their discriminative features. As we know, it is the first time that the gating mechanism and Attention Convolutional Binary Neural Tree are merged into GCN. The main contributions of this paper are summarized as follows:

• We propose a GCN propagation rule in multi-relational graphs that uses Encoder to pass messages in multi-relational Graph which solves the problem of messages passing in high-dimensional feature data in GCN.

• We use an Attention Convolutional Binary Neural Tree based on the attention mechanism to extract the fine-grained features of the image in Encoder. For small local differences, this convolutional binary neural tree structure can learn features of discrimination to distinguish different types of lung diseases.

• We adopt the idea of the partial parameter sharing to reduce the number of parameters in the GGAC which can reduce the complexity of the entire model.

• We design the Gated GCN Block to replace the traditional GCN which solves the problem of weight distribution of different neighbors in the same-order neighborhood, and strengthens the model’s ability to capture spatial information correlation.

• We construct a multi-relational Chest X-rays Graph on the Chest X-rays14 dataset [10] and verify our model. GGAC always has better performance than the previous method in disease identification.