Abstract
The detection and recognition of text instances in camera-captured images or videos generate rich and precise semantic information for interpreting and describing the scene. However, recognizing text in the wild remains a challenging problem despite its value. Apart from the inherent problems in text detection and recognition tasks, knowing the language to be recognized is required in unsupervised forensic applications where multilingual information is frequently employed. This work proposes an unconstrained multilingual text recognition pipeline for scene text detection and recognition. The pipeline consists of multiple text recognition experts contributing to determining the output text sentence. Each expert translates the visual information into a candidate text sentence. Finally, our post-processing model, named PolygloNet, encodes and aggregates all text sentences to generate the optimal text sequence. This model can select the most likely text sequence and correct spelling errors produced in the recognition stage. Experimental results show that our model can produce accurate recognition results on relevant datasets (MLT2017 [2] and MLT2019 [1]).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Overview - ICDAR 2019 robust reading challenge on multi-lingual scene text detection and recognition. https://rrc.cvc.uab.es/?ch=15. Accessed 24 May 2021
Overview - ICDAR2017 competition on multi-lingual scene text detection and script identification. https://rrc.cvc.uab.es/?ch=8. Accessed 24 May 2021
Tesseract OCR. https://github.com/tesseract-ocr/tesseract. Accessed 17 May 2021
Traineddata files for version 4.00+. https://tesseract-ocr.github.io/tessdoc/Data-Files.html. Accessed 24 May 2021
Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection, pp. 9365–9374 (2019)
Chen, X., Jin, L., Zhu, Y., Luo, C., Wang, T.: Text recognition in the wild: a survey. ACM Comput. Surv. 54(2) (2021). https://doi.org/10.1145/3440756
Chen, Z., Yin, F., Zhang, X.Y., Yang, Q., Liu, C.L.: MuLTReNets: multilingual text recognition networks for simultaneous script identification and handwriting recognition. Pattern Recognit. 108, 107555 (2020). https://doi.org/10.1016/j.patcog.2020.107555. https://www.sciencedirect.com/science/article/pii/S0031320320303587
Chernyshova, Y.S., Sheshkus, A.V., Arlazarov, V.V.: Two-step CNN framework for text line recognition in camera-captured images. IEEE Access 8, 32587–32600 (2020). https://doi.org/10.1109/ACCESS.2020.2974051
Du, Y., et al.: PP-OCR: a practical ultra lightweight OCR system. CoRR abs/2009.09941 (2020). https://arxiv.org/abs/2009.09941
Etter, D., Rawls, S., Carpenter, C., Sell, G.: A synthetic recipe for OCR. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 864–869 (2019). https://doi.org/10.1109/ICDAR.2019.00143
Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 369–376. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1143844.1143891
Hasnat, M.A., Chowdhury, M.R., Khan, M.: An open source tesseract based optical character recognizer for Bangla script. In: 2009 10th International Conference on Document Analysis and Recognition, pp. 671–675 (2009). https://doi.org/10.1109/ICDAR.2009.62
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, J., et al.: A multiplexed network for end-to-end, multilingual OCR, pp. 4545–4555 (2021). https://doi.org/10.1109/CVPR46437.2021.00452
Kuang, Z., et al.: MMOCR: a comprehensive toolbox for text detection, recognition and understanding, pp. 3791–3794 (2021). https://doi.org/10.1145/3474085.3478328
Liao, M., Zou, Z., Wan, Z., Yao, C., Bai, X.: Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE Trans. Pattern Anal. Mach. Intell., 1 (2022). https://doi.org/10.1109/TPAMI.2022.3155612
Liu, W., Chen, C., Wong, K.Y.K., Su, Z., Han, J.: Star-Net: a spatial attention residue network for scene text recognition. In: Richard C. Wilson, E.R.H., Smith, W.A.P. (eds.) Proceedings of the British Machine Vision Conference (BMVC), pp. 43.1–43.13. BMVA Press, September 2016. https://doi.org/10.5244/C.30.43. https://dx.doi.org/10.5244/C.30.43
Medina, P., Fidalgo, E., Alegre, E., Alaiz, R., Jáñez-Martino, F., Bonnici, A.: Rectification and super-resolution enhancements for forensic text recognition. Sensors, 32–37 (2020). https://doi.org/10.3390/s20205850
Mindee: docTR: document text recognition (2021). https://github.com/mindee/doctr
Pal, A., Mustafi, A.: Vartani spellcheck - automatic context-sensitive spelling correction of OCR-generated Hindi text using BERT and Levenshtein distance. CoRR abs/2012.07652 (2020). https://arxiv.org/abs/2012.07652
Rostamizadeh, A., Talwalkar, A., DeSalvo, G., Jamieson, K., Li, L.: Efficient hyperparameter optimization and infinitely many armed bandits (2017)
Sakaguchi, K., Duh, K., Post, M., Durme, B.V.: Robsut Wrod reocginiton via semi-character recurrent neural network, pp. 3281–3287 (2017)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017). https://doi.org/10.1109/TPAMI.2016.2646371
Silva, S.M., Jung, C.R.: License plate detection and recognition in unconstrained scenarios. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 593–609. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01258-8_36
Smith, R.: Tesseract blends old and new OCR technology. https://github.com/tesseract-ocr/docs/tree/master/. Accessed 24 May 2021
Tkachenko, I., Gomez-Krämer, P.: Robustness of character recognition techniques to double print-and-scan process. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 09, pp. 27–32 (2017). https://doi.org/10.1109/ICDAR.2017.392
Vaswani, A., et al.: Attention is all you need 30 (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Yang, S., Wang, Y., Chu, X.: A survey of deep learning techniques for neural machine translation. CoRR abs/2002.07526 (2020). https://arxiv.org/abs/2002.07526
Zacharias, E., Teuchler, M., Bernier, B.: Image processing based scene-text detection and recognition with tesseract. CoRR abs/2004.08079 (2020). https://arxiv.org/abs/2004.08079
Zhou, J., Cao, Y., Wang, X., Li, P., Xu, W.: Deep recurrent models with fast-forward connections for neural machine translation. CoRR abs/1606.04199 (2016). http://arxiv.org/abs/1606.04199
Acknowledgement
This work was supported by the European Union’s Horizon 2020 Research and Innovation Program (AIDA Project) under Grant Agreement 883596.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Solé Gómez, À., García Castaño, J., Leškovský, P., Otaegui Madurga, O. (2022). PolygloNet: Multilingual Approach for Scene Text Recognition Without Language Constraints. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13232. Springer, Cham. https://doi.org/10.1007/978-3-031-06430-2_40
Download citation
DOI: https://doi.org/10.1007/978-3-031-06430-2_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06429-6
Online ISBN: 978-3-031-06430-2
eBook Packages: Computer ScienceComputer Science (R0)