Abstract
Text can be widely found in natural scenes. However, it is considerably difficult to detect and recognize the scene text due to its variations and distortions. In this paper, we propose a three-stage bottom-up scene text spotter, including text segmentation, text rectification and text recognition. The text segmentation part adopts a feature pyramid network (FPN) to extract character instances by combining local and global information, then a joint network of FPN and bidirectional long short-term memory is developed to explore the affinity among the isolated characters, which are grouped into character flows. The text rectification part utilizes a spatial transformer network to deal with the complex deformation of the character flows, thus enhancing their readability. Finally, the rectified text is recognized through an attention-based sequence recognition network. Extensive experiments are conducted on several benchmarks, showing that our approach achieves the state-of-the-art performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baek, Y., et al.: Character region attention for text spotting. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 504–521. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_30
Wang, T., et al.: Decoupled attention network for text recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12216–12224 (2020)
Wan, Z., He, M., Chen, H., Bai, X., Yao, C.: TextScanner: reading characters in order for robust scene text recognition. Proc. AAAI Conf. Artif. Intell. 34, 12120–12127 (2020)
Liu, W., Chen, C., Wong, K.Y.: Char-Net: a character-aware neural network for distorted scene text recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7154–7161 (2018)
Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: ASTER: an attentional scene text recognizer with flexible rectification. IEEE Trans. Pattern Anal. Mach. Intell. 41(9), 2035–2048 (2018)
Qi, X., Chen, Y., Xiao, R., Li, C.-G., Zou, Q., Cui, S.: A novel joint character categorization and localization approach for character-level scene text recognition. In: Proceedings of the IEEE International Conference on Document Analysis and Recognition Workshops, pp. 83–90 (2019)
Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. Proc. Int. J. Comput. Vis. 116, 1–20 (2016)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. J. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2016)
He, T., Tian, Z., Huang, W., Shen, C., Qiao, Y., Sun, C.: An end-to-end textspotter with explicit alignment and attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5020–5029 (2018)
Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., Yan, J.: FOTS: fast oriented text spotting with a unified network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5676–5685 (2018)
Yin, F., Wu, Y., Zhang, X., Liu, C.: Scene text recognition with sliding convolutional character models. arXiv preprint arXiv:1709.01727 (2017)
Liao, M., Lyu, P., He, M., Yao, C., Wenhao, W., Bai, X.: Mask text spotter: an end-to-end trainable neural network for spotting text with arbitrary shapes. IEEE Trans. Image Process. 43(2), 532–548 (2021)
He, K.-M., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2961–2969 (2017)
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 936–944 (2017)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. J. Neural Netw. 18(5–6), 602–610 (2005)
Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, K.: Spatial transformer networks. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)
Luo, C., Jin, L., Sun, Z.: MORAN: a multi-object rectified attention network for scene text recognition. Pattern Recogn. 90, 109–118 (2019)
Li, X., Wang, W., Hou, W., Liu, R.-Z., Lu, T., Yang, J.: Shape robust text detection with progressive scale expansion network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9336–9345 (2019)
Deng, D., Liu, H., Li, X., Cai, D.: PixelLink: detecting scene text via instance segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6773–6780 (2018)
Shi, B., Wang, X., Lyu, P., Yao, C., Bai, X.: Robust scene text recognition with automatic rectification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4168–4176 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE (2016)
Karatzas, D.: ICDAR 2013 robust reading competition. In: Proceedings of International Conference on Document Analysis and Recognition, pp. 1484–1493 (2013)
Karatzas, D., et al.: ICDAR 2015 competition on robust reading. In: Proceedings of International Conference on Document Analysis and Recognition, pp. 1156–1160 (2015)
Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2315–2324 (2016)
Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227v4 (2014)
Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: towards accurate text recognition in natural images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5086–5094 (2017)
Cheng, Z., Xu, Y., Bai, F., Niu, Y., Pu, S., Zhou, S.: AON: towards arbitrarily-oriented text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5571–5579 (2018)
Xie, H., Fang, S., Zha, Z., Yang, Y., Li, Y., Zhang, Y.: Convolutional attention networks for scene text recognition. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 15(1s), 3:1–3:17 (2019)
Li, H., Wang, P., Shen, C., Zhang, G.: Show, attend and read: a simple and strong baseline for irregular text recognition. Proc. AAAI Conf. Artif. Intell. 33, 8610–8617 (2019)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61902435, in part by Hunan Provincial Natural Science Foundation of China under Grant 2019JJ50808, and in part by the 111 Project under Grant B18059.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zou, B., Yang, W., Li, K., Huang, E., Liu, S. (2021). Character Flow Detection and Rectification for Scene Text Spotting. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-89029-2_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89028-5
Online ISBN: 978-3-030-89029-2
eBook Packages: Computer ScienceComputer Science (R0)