Soft-Boundary Label Relaxation with class placement constraints for semantic segmentation of the railway environment

Highlights

• Proposes semantic segmentation for railway environmental understanding.

• Soft Boundary Label Relaxation solves misaligned label boundaries in railway images.

• Proposed method is effective for challenging train front-view dataset.

• Proposed method is also effective for general street-view dataset.

Abstract

In this paper, we focus on the challenging task of the semantic segmentation of train front-view images. Managing trackside facilities can be done by using detailed and precise information about the surrounding railway environment. Semantic segmentation enables us to understand the 2D environment, but there is no adequate large-scale dataset available for training a CNN for this purpose. Some attempts have been made to generate pseudo-data from unlabeled sequential frames to compensate for the lack of volume in training data, but the moving speed of trains makes it difficult to apply them directly. We aim to solve this problem by proposing the Soft Boundary Label Relaxation (Soft-BLR) method, which considers label boundaries extending over multiple pixels to cope with more severely distorted pseudo-data and to better train the CNN in the initial training stage. Furthermore, we modify the loss function to penalize inference results based on the distance from the label boundary to solve the misalignment problems of border pixels. Through experimental evaluation, we report that the proposed method outperforms previous methods on not only the semantic segmentation of challenging railway images, but also that of general street-view images.

Introduction

Railways are valued means of transportation due to their speed, capacity, and reliability, and their extension reaches a total of more than a million kilometers around the Globe. To cope with such characteristics, railway operators especially emphasize on the safety and the prevention of accidents. From simple railway signals to more advanced Automatic Train Stop (ATS) systems, various technologies are used to ensure the safety of passengers. However, the collection of geological / geometrical positions and the types of trackside facilities are currently done manually with high human cost, and some railway operators are even unaware of where and what trackside facilities exist along their tracks due to managing problems within different departments. Daily maintenance of such facilities is also essential, yet it is still being done by manual and/or visual inspection. Therefore, a fully automatic technology that can collect data about trackside facilities and can be used for their maintenance is in crucial need for railway operators. To meet such needs, the use of semantic segmentation for railway environment understanding is currently being considered.

Semantic segmentation, a task of allocating a single semantic label to each and every pixel within an image, can be used to understand the surrounding environment in detail. Almost every modern method of semantic segmentation utilizes Convolutional Neural Network (CNN), and thus requires supervised data [3]. For this reason, building an adequate dataset for semantic segmentation is a substantial issue. A typical pixel-level manual annotation of an image takes more than an hour [4]. Training a CNN model generally requires a massive volume of training data, and constructing a large-scale dataset for every application is unrealistic. Although domain adaptation has been studied to transfer training results to similar domains, such as synthetic to real-world data [10], this approach cannot be applied across dissimilar domains like from street environment to the railway environment.

To cope with such lack of sufficient training data, Zhu et al. [18] originally proposed joint image-label propagation to generate pseudo-data using a small number of labeled images and neighboring sequential unlabeled images for the street-view image domain. They also introduced Boundary Label Relaxation (BLR) to cope with distorted training data generated by joint image-label propagation. In joint image-label propagation, pseudo-data are generated by transforming both an image and its label using densely calculated optical flows between sequential frames. During this transformation, multiple labels can propagate to a single pixel location, thus making the class boundary ambiguous. BLR is introduced to take into account such ambiguity during the training of a CNN. They were able to augment the Cityscapes dataset [4] to eleven times its original size and train a CNN effectively using this method, but the following problems remain:

• Large-scale augmentation is only possible when the dataset is taken at a high frame-rate and with small camera movement. With data taken from cameras mounted on trains, high operation speed of trains makes it difficult to propagate the labels for generating pseudo-data. Distant pixels can be propagated to a single location, increasing the label ambiguity. However, the original method [18] only considers BLR for areas within distance 1 of a label boundary.

• BLR modifies the label space to allow multiple labels as ground truth in a single pixel location (Fig. 1). This means that inferences with wrong label alignment may be considered as being correct.

To tackle these problems, based on the fact that label boundary distortions of more than a single pixel appear when propagating labels of railway images, we propose a “Soft-BLR” that considers a larger width at the label boundary. We introduce a novel loss function of the CNN to penalize inference results based on the distance from the relaxed label boundary to solve the misalignment problems existent in the original BLR. This loss function also enables a smoother training of the CNN, as close misclassifications will have lower loss values than those of distant misclassifications. These efforts contribute to accurate semantic segmentation of railway environments, even in cases where the original annotations for training are barely available. We demonstrate the effectiveness of the proposed method through experimental analysis, and discuss future applications and possible further improvements.