Skip to main content
Springer Nature Link
Log in

Text detection and localization in natural scene images based on text awareness score

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Text detection & localization plays an essential role in finding the textual information from natural scene images that can be used in robot navigation, license plate detection, and wearable applications. In this work, we present text detection and localization approach based upon a novel text awareness model that encompasses an improved fast edge preserving and smoothing Maximum Stable Extremal Region (FEPS-MSER) algorithm which uses the fast guided filter to separate the interconnected characters efficiently by removing the mixed pixels around the edges of blurred images. The fast guided filter takes less execution time as compared to other edge-smoothing filters. The combination of five independent and class determining facets namely stroke width deviation, 8-histogram of edge gradients, color variation, occupation ratio, and occupy rate convex area is proposed to differentiate between text and non-text components. The probability of a component to be text is based on Text Awareness Score (TAS) that is calculated by fusing these facets in Naive Bayes using the observation possibility and prior probability of text & non-text components. Naïve Bayes classifier helps in accurate and fast determination of the text awareness score and thus helps in the classification of text & non-text components with the help of graph cut algorithm. The text components have been grouped by using the mean-shift clustering algorithm which is a non-parametric technique and does not require the initial knowledge of clusters. The proposed method achieves improved results concerning precision, recall, and f-measure on the ICDAR benchmark datasets for natural scene images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. more examples shown in Section 6.9

  2. explained in detail in Section 6.8

  3. In Fig. 10 the text detection is shown by using SWD facet alone to show its comparison with SWV [36]. The final results are calculated by combination of all five facets using naive bayes in Section 4.4.

  4. In Fig. 12 the text detection is performed by using 8HOGE facet alone to show its comparison with eHOG [36]. The final results are calculated by the combination of all five facets using Naive Bayes in Section 4.4.

  5. http://liris.cnrs.fr/christian.wolf/software/deteval/index.html

References

  1. Bouakkaz M, Ouinten Y, Loudcher S, Fournier-Viger P (2018) Efficiently mining frequent itemsets applied for textual aggregation. Appl Intell 48(4):1013–1019

    Article  Google Scholar 

  2. Zhu Y, Yao C, Bai X (2016) Scene text detection and recognition: Recent advances and future trends. Front Comput Sci 10(1):19–36

    Article  Google Scholar 

  3. Zhang H, Zhao K, Song YZ, Guo J (2013) Text extraction from natural scene image: A survey. Neurocomputing 122:310–323

    Article  Google Scholar 

  4. Jung K, Kim KI, Jain AK (2004) Text information extraction in images and video: a survey. Pattern Recog 37(5):977–997

    Article  Google Scholar 

  5. Unar S, Hussain A, Shaikh M, Memon KH, Ansari MA, Memon Z (2018) A study on text detection and localization techniques for natural scene images. IJCSNS 18(1):100

    Google Scholar 

  6. Lucas SM, Panaretos A, Sosa L, Tang A, Wong S, Young R (2003) Icdar 2003 robust reading competitions. In: ICDAR, Citeseer, vol 2003, p 682

  7. Shahab A, Shafait F, Dengel A (2011) Icdar 2011 robust reading competition challenge 2: Reading text in scene images. In: 2011 International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp 1491–1496

  8. Karatzas D, Shafait F, Uchida S, Iwamura M, i Bigorda LG, Mestre SR, Mas J, Mota DF, Almazan JA, de las Heras LP (2013) Icdar 2013 robust reading competition. In: 2013 12th International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp 1484–1493

  9. Yu C, Song Y, Zhang Y (2016) Scene text localization using edge analysis and feature pool. Neurocomputing 175:652–661

    Article  Google Scholar 

  10. da Silva BLS, Ciarelli PM (2016) Edge detection and confidence map applied to identify textual elements in images

  11. Lee S, Cho MS, Jung K, Kim JH (2010) Scene text extraction with edge constraint and text collinearity. In: 2010 20th International Conference on Pattern Recognition (ICPR), IEEE, pp 3983–3986

  12. Bai B, Yin F, Liu CL (2013) Scene text localization using gradient local correlation. In: 2013 12th International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp 1380–1384

  13. Zhang J, Kasturi R (2010) Text detection using edge gradient and graph spectrum. In: 2010 20th International Conference on Pattern Recognition (ICPR), IEEE, pp 3979–3982

  14. Epshtein B, Ofek E, Wexler Y (2010) Detecting text in natural scenes with stroke width transform. In: 2010 IEEE conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 2963–2970

  15. Yi C, Tian Y (2011) Text string detection from natural scenes by structure-based partition and grouping. IEEE Trans Image Process 20(9):2594–2605

    Article  MathSciNet  MATH  Google Scholar 

  16. Li Y, Lu H (2012) Scene text detection via stroke width. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp 681–684

  17. Zhang G, Kai H, Zhang B, Fu H, Zhao J (2017) A natural scene text extraction method based on the maximum stable extremal region and stroke width transform. J Xi’an Jiaotong Univ 1:021

    Google Scholar 

  18. Wu H, Zou B, Zhao YQ, Guo J (2017) Scene text detection using adaptive color reduction, adjacent character model and hybrid verification strategy. Vis Comput 33(1):113–126

    Article  Google Scholar 

  19. Matas J, Chum O, Urban M, Pajdla T (2004) Robust wide-baseline stereo from maximally stable extremal regions. Image Vis Comput 22(10):761–767

    Article  Google Scholar 

  20. Pan YF, Hou X, Liu CL (2011) A hybrid approach to detect and localize texts in natural scene images. IEEE Trans Image Process 20(3):800–813

    Article  MathSciNet  MATH  Google Scholar 

  21. Guan L, Chu J (2017) Natural scene text detection based on swt, mser and candidate classification. In: 2017 2nd International Conference on Image, Vision and Computing (ICIVC), IEEE, pp 26–30

  22. Feng Y, Song Y, Zhang Y (2016) Scene text detection based on multi-scale swt and edge filtering. In: 2016 23rd International Conference on Pattern Recognition (ICPR), IEEE, pp 645–650

  23. Jiang M, Cheng J, Chen M, Ku X (2018) An improved text localization method for natural scene images. J Phys Conf Series 960(1):012027

    Google Scholar 

  24. Baran R, Partila P, Wilk R (2018) Automated text detection and character recognition in natural scenes based on local image features and contour processing techniques. In: International conference on intelligent human systems integration, Springer, pp 42–48

  25. Ghanei S, Faez K (2017) A robust approach for scene text localization using rule-based confidence map and grouping. Int J Pattern Recog Artif Intell 31(03):1753002

    Article  Google Scholar 

  26. Wei Y, Zhang Z, Shen W, Zeng D, Fang M, Zhou S (2017) Text detection in scene images based on exhaustive segmentation. Signal Process Image Commun 50:1–8

    Article  Google Scholar 

  27. Joan SF, Valli S (2017) An enhanced text detection technique for the visually impaired to read text. Inf Syst Front 19(5):1039–1056

    Article  Google Scholar 

  28. Šarić M (2017) Scene text segmentation using low variation extremal regions and sorting based character grouping. Neurocomputing 266:56–65

    Article  Google Scholar 

  29. Guo M, Yi Y, Liu J, Li Y (2016) Scene text segmentation method based on mser and mlbp. In: China Academic Conference on Printing & Packaging and Media Technology, Springer, pp 305–310

  30. Nguyen K, Thanh ND (2016) Scene text detection based on structural features. In: 2016 International Conference on Computer, Control, Informatics and its Applications (IC3INA), IEEE, pp 48–53

  31. Zheng Y, Liu H, Liu J, Li Q, Li G (2016) Robust scene text detection based on color consistency. In: Eighth International Conference on Digital Image Processing (ICDIP 2016), International Society for Optics and Photonics, vol 10033, p 100334Q

  32. Wu H, Zou B, Yq Zhao, Chen Z, Zhu C, Guo J (2016) Natural scene text detection by multi-scale adaptive color clustering and non-text filtering. Neurocomputing 214:1011–1025

    Article  Google Scholar 

  33. Gomez L, Karatzas D (2016) A fast hierarchical method for multi-script and arbitrary oriented scene text extraction. International Journal on Document Analysis and Recognition (IJDAR) 19(4):335–349

    Article  Google Scholar 

  34. Fabrizio J, Robert-Seidowsky M, Dubuisson S, Calarasanu S, Boissel R (2016) Textcatcher: a method to detect curved and challenging text in natural scenes. International Journal on Document Analysis and Recognition (IJDAR) 19(2):99–117

    Article  Google Scholar 

  35. Wang X, Song Y, Zhang Y, Xin J (2015) Natural scene text detection with multi-layer segmentation and higher order conditional random field based analysis. Pattern Recogn Lett 60:41–47

    Article  Google Scholar 

  36. Li Y, Jia W, Shen C, van den Hengel A (2014) Characterness: An indicator of text in the wild. IEEE Trans Image Process 23(4):1666–1677

    Article  MathSciNet  MATH  Google Scholar 

  37. Koo HI, Kim DH (2013) Scene text detection via connected component clustering and nontext filtering. IEEE Trans Image Process 22(6):2296–2305

    Article  MathSciNet  MATH  Google Scholar 

  38. Felhi M, Bonnier N, Tabbone S (2012) A skeleton based descriptor for detecting text in real scene images. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp 282–285

  39. Neumann L, Matas J (2012) Real-time scene text localization and recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, IEEE, pp 3538–3545

  40. Yi C, Tian Y (2012) Localizing text in scene images by boundary clustering, stroke segmentation, and string fragment classification. IEEE Trans Image Process 21(9):4256–4268

    Article  MathSciNet  MATH  Google Scholar 

  41. Meng Q, Song Y (2012) Text detection in natural scenes with salient region. In: 2012 10th IAPR international workshop on Document Analysis Systems (DAS), IEEE, pp 384–388

  42. Tian C, Xia Y, Zhang X, Gao X (2017) Natural scene text detection with mc–mr candidate extraction and coarse-to-fine filtering. Neurocomputing 260:112–122

    Article  Google Scholar 

  43. Neumann L, Matas J (2010) A method for text localization and recognition in real-world images. In: Asian Conference on Computer Vision, Springer, pp 770–783

  44. Chen H, Tsai SS, Schroth G, Chen DM, Grzeszczuk R, Girod B (2011) Robust text detection in natural images with edge-enhanced maximally stable extremal regions. In: 2011 18th IEEE International Conference on Image Processing (ICIP), IEEE, pp 2609–2612

  45. Donoser M, Bischof H (2006) Efficient maximally stable extremal region (mser) tracking. In: 2006 IEEE computer society conference on computer vision and pattern recognition, IEEE, vol 1, pp 553–560

  46. Raju A et al (2013) A comparative analysis of histogram equalization based techniques for contrast enhancement and brightness preserving

  47. Sun L, Huo Q, Jia W, Chen K (2015) A robust approach for text detection from natural scene images. Pattern Recog 48(9):2906–2920

    Article  Google Scholar 

  48. He K, Sun J (2015) Fast guided filter. arXiv:150500996

  49. He K, Sun J, Tang X (2010) Guided image filtering. In: European conference on computer vision, Springer, pp 1–14

  50. Zhang Q, Xu L, Jia J (2014) 100+ times faster weighted median filter (wmf). In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 2830–2837

  51. Yao C, Bai X, Liu W, Ma Y, Tu Z (2012) Detecting texts of arbitrary orientations in natural images. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 1083–1090

  52. Abdi H (2010) Coefficient of variation. Encycl Res Des 1:169–171

    Google Scholar 

  53. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, IEEE, vol 1, pp 886–893

  54. Majtey A, Lamberti P, Prato D (2005) Jensen-shannon divergence as a measure of distinguishability between mixed quantum states. Phys Rev A 72(5):052310

    Article  Google Scholar 

  55. Klein DA, Frintrop S (2011) Center-surround divergence of feature statistics for salient object detection. In: 2011 IEEE International Conference on Computer Vision (ICCV), IEEE, pp 2214–2219

  56. Wang Q, Lu Y, Sun S (2015) Text detection in nature scene images using two-stage nontext filtering. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp 106–110

  57. Gonzalez A, Bergasa LM, Yebes JJ, Bronte S (2012) Text location in complex images. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp 617–620

  58. Al-khurayji R, Sameh A (2017) An effective arabic text classification approach based on kernel naive bayes classifier. Int J Artif Intell Appl 8(6):01–10

    Google Scholar 

  59. Xiang ZL, Yu XR, Kang DK (2016) Experimental analysis of naïve bayes classifier based on an attribute weighting framework with smooth kernel density estimations. Appl Intell 44(3):611–620

    Article  Google Scholar 

  60. Kim HK, Kim M (2016) Model-induced term-weighting schemes for text classification. Appl Intell 45 (1):30–43

    Article  Google Scholar 

  61. Rish I et al (2001) An empirical study of the naive bayes classifier. In: IJCAI 2001 workshop on empirical methods in artificial intelligence, IBM New York, vol 3, pp 41–46

  62. McCallum A, Nigam K et al (1998) A comparison of event models for naive bayes text classification. In: AAAI-98 workshop on learning for text categorization, Citeseer, vol 752, pp 41–48

  63. Feng G, Guo J, Jing BY, Sun T (2015) Feature subset selection using naive bayes for text classification. Pattern Recogn Lett 65:109–115

    Article  Google Scholar 

  64. Tang B, Kay S, He H (2016) Toward optimal feature selection in naive bayes for text categorization. arXiv:160202850

  65. Singh A, Halgamuge MN, Lakshmiganthan R (2017) Impact of different data types on classifier performance of random forest, naive bayes, and k-nearest neighbors algorithms. Int J Adv Comput Sci Appl 8(12):1–10

    Google Scholar 

  66. Miralles-Pechuán L, Rosso D, Jiménez F, García JM (2017) A methodology based on deep learning for advert value calculation in cpm, cpc and cpa networks. Soft Comput 21(3):651–665

    Article  Google Scholar 

  67. Udomsak N (2015) How do the naive bayes classifier and the support vector machine compare in their ability to forecast the stock exchange of thailand? arXiv:151108987

  68. Colas FPR et al (2009) Data mining scenarios for the discovery of subtypes and the comparison of algorithms Leiden Institute of Advanced Computer Science (LIACS), Faculty of Science, Leiden University

  69. Gonzalez A, Bergasa LM, Yebes JJ (2014) Text detection and recognition on traffic panels from street-level imagery using visual appearance. IEEE Trans Intell Transp Syst Mag 15(1):228–238

    Article  Google Scholar 

  70. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2 (2):121–167

    Article  Google Scholar 

  71. Shi H, Liu Y (2011) Naïve bayes vs. support vector machine: resilience to missing data. In: International Conference on Artificial Intelligence and Computational Intelligence, Springer, pp 680–687

  72. Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell 23(11):1222–1239

    Article  Google Scholar 

  73. Chong HY, Gortler SJ, Zickler T (2008) A perception-based color space for illumination-invariant image processing. In: ACM Transactions on Graphics (TOG), ACM, vol 27, p 61

  74. Fukunaga K, Hostetler L (1975) The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans Inf Theory 21(1):32–40

    Article  MathSciNet  MATH  Google Scholar 

  75. Carreira-Perpinán MA (2015) A review of mean-shift algorithms for clustering. arXiv:150300687

  76. Vedaldi A, Fulkerson B (2010) Vlfeat: An open and portable library of computer vision algorithms. In: Proceedings of the 18th ACM international conference on Multimedia, ACM, pp 1469–1472

  77. Wolf C, Jolion JM (2006) Object count/area graphs for the evaluation of object detection and segmentation algorithms. International Journal of Document Analysis and Recognition (IJDAR) 8(4):280–296

    Article  Google Scholar 

  78. Lucas SM (2005) Icdar 2005 text locating competition results. In: Eighth international conference on Document Analysis and Recognition, 2005. Proceedings, IEEE, pp 80–84

  79. Pal C, Chakrabarti A, Ghosh R (2015) A brief survey of recent edge-preserving smoothing algorithms on digital images. arXiv:150307297

  80. Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images 1998 Sixth international conference on computer vision, IEEE, pp 839–846

  81. Feng Y, Song Y, Zhang Y (2015) Scene text localization using extremal regions and corner-hog feature. In: 2015 IEEE international conference on Robotics and Biomimetics (ROBIO), IEEE, pp 881–886

  82. Wang R, Sang N, Gao C (2015) Text detection approach based on confidence map and context information. Neurocomputing 157:153–165

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by UPE-II, Jawaharlal University, New Delhi, India.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, NSIT, New Delhi, India

    Rituraj Soni & Bijendra Kumar

  2. School of Computer and Systems Science, JNU, New Delhi, India

    Satish Chand

Authors
  1. Rituraj Soni
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Bijendra Kumar
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Satish Chand
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rituraj Soni.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Soni, R., Kumar, B. & Chand, S. Text detection and localization in natural scene images based on text awareness score. Appl Intell 49, 1376–1405 (2019). https://doi.org/10.1007/s10489-018-1338-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-018-1338-4

Keywords