A scene segmentation algorithm combining the body and the edge of the object

Highlights

• This article proposes the BEJNet network structure which combines the main body and the edge of the object for better feature extraction. On the basis of the semantic flow feature alignment module, the U-shaped body context information extraction module with residual connections is designed to comprehensively use the object body local and global context information while keeping the flow model training stable.

• This article proposes an edge attention module, which uses high-level information to generate edge features containing semantic information and combines low-level edge features guided by the global pooling module to refine the edge features of objects. So, the segmentation effect of the object edges is improved.

• Experimental results of the proposed method on many classic network structures such as FCN, PSPNet, DeepLabv3+ and SFNet structures, which improves the mIoU of semantic segmentation with tiny parameters. In addition, we also conduct tests on several classic scene datasets of Cityscapes, CamVid and KITTI, which indicates that our proposed method have reached good results.

Abstract

Scene segmentation is a very challenging task where convolutional neural networks are used in this field and have achieved very good results. Current scene segmentation methods often ignore the internal consistency of the target object, and lack to make full use of global and local context information which leads to the situation of object misclassification. In addition, most of the previous work focused on the segmentation of the main part of the object, however, there are few researches on the quality of the object edge segmentation. In this article, based on the use of flow information to maintain body consistency, the context feature extraction module is designed to fully consider the global and local body context information of the target object, refining the rough feature map in the intermediate stage. So, the misclassification of the target object is reduced. Besides, in the proposed edge attention module, the low-level feature map guided by the global feature and the edge feature map with semantic information obtained by intermediate process are connected to obtain more accurate edge detail information. Finally, the segmentation quality that contains the body part of the noise and the edge details can be improved. This paper not only conducts experiments on the classic FCN, PSPNet, and DeepLabv3+ several mainstream network architectures, but also on the real-time SFNet network structure proposed last year, and the value of mIoU in object and boundary is improved to verify the effectiveness of the method proposed in this paper. Moreover, in order to prove the robustness of the experiment, we conduct experiments on three complex scene segmentation data sets of Cityscapes, CamVid, and KiTTi, and obtained mIoU values of 80.52% on the Cityscapes validation data set, and 71.4%, 56.53% on the Camvid and KITTI test data set, which shows better results when compared with most of the state-of-the-art methods.

Introduction

Scene segmentation is to assign a category label to each pixel of the object in the image based on semantic information. It is the main content of the current field of vision research and has a very high theoretical significance for promoting the development of real scenes. At present, scene segmentation has been widely used in unmanned driving (Feng et al., 2020), indoor navigation (Wang et al., 2018), video surveillance (Franklin and Dabbagol, 2020) and many other fields.

Although the full convolutional neural network has achieved great success in semantic segmentation, however, achieving accurate segmentation and distinguishing targets in complex scenes is still an important research direction currently. On the one hand, there will be noise inside the target to be segmented in the scene image, the internal consistency of the object is difficult to maintain, besides, the object category is easy to be misclassified when the global context information is lacking (Zhou et al., 2019); on the other hand, while the deep network captures deep semantic information, the down-sampling operation will lose a lot detail information on the edge of the object (Chen et al., 2019), so, it is difficult to segment small targets and object edges. For this reason, how to effectively capture the global and local context relationships between multiple objects in a picture to reduce category misclassification while maintaining internal consistency of objects, and improve the performance of object edge segmentation has become a key point in the research direction of scene segmentation.

In response to the first problem, the pyramid pooling module, the atrous convolution, etc. have been proposed to further expand the receptive field while capturing multi-scale features successively; for the second problem, in order to make deep and high-level features contain strong semantic information, but also retain the rich detailed information of the object, the high-level and low-level feature cascade (Pang et al., 2019) or connect (Zhao et al., 2018) is a common way. This is no doubt that these methods improve the performance of semantic segmentation. Although the multi-scale feature extraction module helps to capture objects of different scales, it cannot make full use of the relationship between objects and objects and the global image, it also lacks the attention to the consistency of the internal pixels of the object. In addition, the common low-level and high-level feature fusion methods lack the interactive use of the information that exists in the body and boundary of the object in the picture which is important for improving the performance of image edge segmentation (Takikawa et al., 2019). It can be viewed from Fig. 1 that the edges obtained from the label image contain whether they are edges and edge category information.

In this article, on the basis of the idea of semantic flow in scene segmentation (Li et al., 2020), we design the body and edge joint network abbreviated as BEJNet. After extracting high-level features from the deep Resnet network structure, the consistency of the pixel values inside the object can be maintained based on the object flow information. We believe that maintaining the saliency feature map with the same pixels within the object body is more conducive to learning context. Based on the designed body context feature capture module, the object category can be accurately identified and the misclassification of objects can be reduced. On the other hand, the edges containing semantic information obtained by subtracting the main features and the low-level edge features guided by the global features can be connected to better capture the edge features. After connecting the body feature map guided by local and global context with edge map obtained from edge attention module, the final semantic segmentation result achieved is more accurate. The proposed module is lightweight and can be migrated to any network structure with simple modifications to form an end-to-end semantic segmentation network. We do ablation experiments on the Cityscapes training and validation datasets on the FCN network structure, and conduct experiments on multiple scene data sets and multiple classic network structures. Finally, the value of mIoU on the Cityscapes obtained from the Cityscapes validation dataset achieves 80.52%. Besides, the method proposed in this paper obtains an mIoU value of 71.4% in Camvid, and the value of mIoU is 56.53% on the KITTI test datasets.

The main contributions of this work are:

• This article proposes the BEJNet network structure which combines the main body and the edge of the object for better feature extraction. On the basis of the semantic flow feature alignment module, the U-shaped body context information extraction module with residual connections is designed to comprehensively use the object body local and global context information while keeping the flow model training stable;

• This article proposes an edge attention module, which uses high-level information to generate edge features containing semantic information and combines low-level edge features guided by the global pooling module to refine the edge features of objects. So, the segmentation effect of the object edges is improved;

• The proposed method experimentes on many classic network structures such as FCN, PSPNet, DeepLabv3+ and SFNet structures recently, which improves the mIoU of semantic segmentation with tiny parameters further. In addition, we are also working on several classic scene data of Cityscapes, CamVid and KITTI experiments on the dataset, all of which have reached good results.