Elite Loss for scene text detection

Abstract

Many scene text detection approaches generate foreground segmentation maps to detect the text instances. In these methods, usually all the pixels within the bounding box regions of the text are equally treated as foreground during the training process. However, different from the general object segmentation problem, we argue that not all the pixels across the text bounding box region contribute equally for locating the text instance. Specifically, some in-box not-on-stroke pixels even degrade the detection performance. Moreover, for the segmentation based methods with a regression step applied to predict the corresponding bounding box on each pixel, not all the pixels need to be fully trained to predict foreground texts. Therefore, in this paper, we propose Elite Loss, which is intended to down-weight the contributions of the in-box not-on-stoke pixels while paying more attention to the on-stoke pixels. Furthermore, we design a segmentation-based method to validate the effectiveness of the proposed Elite Loss. Extensive experiments demonstrate that our methods achieve the state-of-the-art results on all three challenging datasets, with the F-score of 0.855 on ICDAR2015, 0.425 on COCO-Text, and 0.819 on MSRA-TD500.

Introduction

Scene text detection has attracted more and more attention for its important role in many computer vision tasks. It aims at localizing the text with bounding boxes of words or text lines. Various methods have been proposed to tackle this problem [1], [2], [3], [4], [5]. The main challenges of scene text detection are the large variety of texts in scales, layouts, fonts, and orientations, as well as the cluttered background which is easily confused with text. Traditional text detection approaches are mostly bottom-up [6], [7], [8]. Lots of hand-craft features are designed to distinguish text from background regions. But these methods perform poorly on complex scenes.

Recently, many deep learning based methods are proposed to detect texts. Some methods are evolved from general object detection methods like SSD [9] or Faster RCNN [10]. They utilized the reference boxes to detect text, such as TextBoxes [11], TextBoxes++ [12], SegLink [1], RRPN [5], R²CNN [13], RRD [14] FSTN [15], etc. These methods shows some improvement over traditional approaches, but they cannot deal with the multi-oriented texts well, as discussed in [4].

Another category of deep learning based methods are inspired by FCN [16], which is widely used in various segmentation tasks. Zhang et al. [17] and He et al. [18] used FCN to locate raw text regions, but they need complicated post-processing steps. Lyu et al. [19] uses corners and position-sensitive segmentation maps to segment each text instances. Liu et al. [20] proposed MCN, in which they designed the Markov Clustering Network to group the segmented foreground pixels into text instances. Recently, some methods such as EAST [2] and Direct Regression [4], [21] are proposed. They regress a bounding box on each pixel’s location, in an end-to-end way. We call these methods segmentation based methods because they detect text in a segmentation-like style.

The segmentation based methods detect the text instances by pixel-wise predictions. However, due to the lack of fine-grained text segmentation annotations, they usually assume all the pixels inside the text’s bounding box as foreground. This is different from the general object segmentation task, where the sub-regions in the target segmentation map have consistent counters with the original objects. We consider that the use of the raw ground truth in the segmentation step is a trade-off caused by the lack of ground truth segmentation maps of text strokes. However, this leads to the optimization error in the backward propagation process of the networks. Fig. 1(c) shows an output score map of a segmentation based detector [2] after the first iteration when it is being fine-tuned from ImageNet-pretrained models. It can be seen that the detector has high activations on the stroke areas and relatively low activations on the smooth areas outside the strokes. Obviously, the pixels on the in-box outside-stroke areas cannot be easily distinguished from the outside-box background pixels for the similar appearance. This means that pixels on the strokes capture the most distinctive characteristics of the text and thus they are easy to learn, comparing to the pixels on the smooth outside-stroke region. For example, from Fig. 1 (a–c), it can be seen that the pixel A has a stronger response at its corresponding location on the score map than the pixel B after the first iteration.

This paper takes example for the category of segmentation based methods which are with an additional regression step [2], [4], to further inspect the above phenomenon. These methods perform the classification and bounding box regression on each pixel of the output segmentation maps to detect the text instances. Each pixel outputs one text instance bounding box if it is classified as positive. We can conclude that the pixels of the score map are the basic elements for the detection task and act independently of each other. In this paper, we call these basic elements as predicting units to better describe their roles. These methods treat all the predicting units of the same corresponding text instance equally. In this case, the total training loss of all the predicting units of the text instance may be easily dominated by those in-box not-on-stroke predicting units, for they usually have high loss value. However, the essential on-stroke predicting units are less considered, for they usually have small loss value. Meanwhile, since many of the not-on-stroke predicting units tend to have similar appearances with predicting units on the background (e.g., the unit at the location of pixel B in Fig. 1), forcing the network to distinguish the hard not-on-stroke predicting units as foreground may lead to many false detections on the background region.

From another view, for the predicting units of the same corresponding text instance, their tasks are duplicated. That is they are all able to find the same text instance independently. Thus it is unnecessary to cost much on those not-on-stroke but hard predicting units. The detector should focus on predicting units that represent the text instance better to learn a robust text detector.

It is worth to note that the not-on-stroke but hard predicting units are not the same as the hard samples chosen from all training samples in the hard example mining [22] or bootstrapping [23] procedure that should be considered more to benefit the performance. The hard predicting units here are the positive samples which are relatively hard but unnecessary to learn. These not-on-stroke but hard predicting units can also be regarded as units with noise labels in a sense, because their appearance is similar to the background and it is unreasonable to consider them the typical positive samples during training.

As analyzed above, we consider the on-stroke predicting units capture the instinct characteristics of text regions, so we call them as elite predicting units. In this paper, we propose a new loss re-weighting strategy to train the detector. We re-weight the classification losses of the prediction units to automatically lower the not-on-stroke but hard predicting units’ contributions to the loss during training. This helps the detector to focus on learning better features to correctly classify the elite on-stroke predicting units on the foreground. We call this reweighed loss as Elite Loss, for the reason that it focuses more on those elite predicting units (i.e., the on-stroke prediction units), and pays less attention to the noisy predicting units, i.e. not-on-stroke predicting units. The Elite Loss is flexible in its specific forms and it is effective. It improves our self-built baseline detector significantly and reaches the new state-of-art on various benchmarks. In Fig. 1, we use the method of [24] to show the difference about effective receptive fields of the same pixel of score map at the location of point A with and without the Elite Loss. We can conclude that the Elite Loss makes the effective receptive field more concentrated. This is beneficial to the pixel-wise classification task, since the surrounding noises are largely suppressed.

The contributions of this paper are listed as follows:

• We propose the Elite Loss in segmentation based text detection networks which have a regression step, by down-weighting the contributions of the in-box not-on-stoke pixels to improve the training performance.

• To demonstrate the effectiveness and the flexibility of Elite Loss, we design two specific forms of Elite Loss and evaluate them in the task of text detection.

• With the Elite loss integrated into the segmentation based text detector which has a regression step, we achieve state-of-the-art results on various datasets.