Unattached irregular scene text rectification with refined objective

Abstract

Recently, text recognition tasks have reached a high level with deep learning-based methods. The techniques are widely applied in different fields, and nowadays most researchers aim to build an effective approach to deal with irregular text in scene images. In this work, we propose a GAN-based framework to rectify scene text with rotation, curving, or other distortions. Unlike previous rectification modules that rely on the recognition networks, our model can be utilized either as an independent model or an extra component. Therefore, annotations of the text content are not required to train the model. And we utilize a refined training objective with the proposed sample loss, which is able to effectively control pixels in the output images that are supposed to be sampled from input ones. Experiments on public benchmarks demonstrate the effectiveness of our method. The code will be publicly available on github soon.

Introduction

Text in scene images usually convey critical information for image analysis, so text recognition is of great significance in the landscape of computer vision. Previous methods usually need to segment the characters in a word at first, and then recognize each character. But over recent years, end-to-end techniques based on deep learning have been extensively applied, and CNN-based methods for text recognition have also sprung up [1], [2], [3], [4]. Excellent achievements have been achieved, and these works mostly have a framework with three modules: convolution layers, recurrent layers, and the transcription layer. This scheme eliminates the demands of the segmentation of characters, which avoids the error accumulating problems. However, there is still a premise that is the arrangement of the text is basically from left to right and there is no severe distortion. If the text is rotated, curved, or distorted by a shift in perspective, this procedure may fail to recognize the words.

In order to deal with irregular text in scene images, researchers were devoted to solving the issue. Two mainstream ideas were proposed: to improve the modules of networks, or to rectify the image before recognition. The former kind of approaches usually focused on the architecture of the networks. They aimed to improve the shape of receptive fields of the convolution layers or strategies of the attention mechanism [5]. The latter one tried to find effective rectification modules, and utilize them in the front of recognition networks [2], [3]. Baek et al. [4] summarized most of the models for text recognizing with different combinations of each module, and they discovered that spatial transformer networks (STN) [6] have been widely utilized to process irregular text before recognition. The best combination they proposed also relies on STN [6] as the pre-processing step. Though all of the methods achieved excellent performances, to utilize STN [6] seems more adaptable owing that they are able to be involved in any state-of-the-art recognition networks in the future.

Whereas there are some advantages of the methods to rectify text images before recognition, rectification modules are scarce to build a robust model facing various applications. And it is disturbing that most of the rectification models need to be trained with the recognition network simultaneously. Though there is little influence on recognizing, this feature restricts the scenarios of applications because the training process requires annotations of the text contents. In some circumstances, the rectification is applied for better showing rather than recognizing, so the redundant annotations will cause a waste of time-consuming. To mitigate this issue, in this paper, we propose an unattached scene text rectification model to deal with irregular text images. It is based on Generative Adversarial Networks (GAN) [7] which can be trained independently. The generator has an encoder-decoder framework, and ResNeXt-50 [8] is utilized as the encoder. The decoder consists of some transposed convolutions with Residual Blocks (ResBlock) [9] and skip connections [10] involved. Then we also utilize a refined training objective to optimize the model, including a new proposed sample loss which is highly effective in the domination of pixels in output images that are supposed to be sampled from the input one. Experiments on public datasets demonstrate the effectiveness of our proposed method. To summarize, there are three main contributions of this work:

• We propose an unattached model for scene text rectification. It can be trained without recognition modules or the annotations of text content and can be utilized either as an independent model or as an extra module in the front of recognition models.

• A refined training objective is presented to build a robust and effective model. Four loss functions are involved, and the proposed sample loss shows competitive effectiveness in this rectification task.

• The code and dataset will be publicly available soon. To facilitate the reference of researches for more progress, we will upload related materials in the future.

The rest of this paper is organized as below. Section 2 introduces the related works. Section 3 describes the proposed method in detail, including the network architecture and the training objective. The experimental results are given in Section 4, and the analysis is provided. Finally, Section 5 discusses our method and gives the conclusion of this paper.