Skip to main content
Springer Nature Link
Log in

Text box proposals for handwritten word spotting from documents

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

In this article, we propose a new approach to segmentation-free word spotting that is based on the combination of three different contributions. Firstly, inspired by the success of bounding box proposal algorithms in object recognition, we propose a scheme to generate a set of word-independent text box proposals. For that, we generate a set of atomic bounding boxes based on simple connected component analysis that are combined using a set of spatial constraints in order to generate the final set of text box proposals. Secondly, an attribute representation based on the Pyramidal Histogram of Characters (PHOC) is encoded in an integral image and used to efficiently evaluate text box proposals for retrieval. Thirdly, we also propose an indexing scheme for fast retrieval based on character n-grams. For the generation of the index a similar attribute space based on a Pyramidal Histogram of Character N-grams (PHON) is used. All attribute models are learned using linear SVMs over the Fisher Vector representation of the word images along with the PHOC or PHON labels of the corresponding words. We show the performance of the proposed approach in both tasks of query-by-string and query-by-example in standard single- and multi-writer data sets, reporting state-of-the-art results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. We understand by segmentation-free the ability of the method to search a word in a whole non-segmented page as opposed to a segmentation-based scenario where retrieval is performed on segmented word images.

References

  1. Plamondon, R., Srihari, S.: On-line and offline handwriting recognition: a comprehensive survey. IEEE Trans. PAMI 22(1), 63–84 (2000)

    Article  Google Scholar 

  2. Vinciarelli, A., Bengio, S., Bunke, H.: Offline recognition of unconstrained handwritten texts using HMMs and statistical language models. IEEE Trans. PAMI 26, 709–720 (2004)

    Article  Google Scholar 

  3. Rodríguez-Serrano, J., Perronnin, F.: Local gradient histogram features for word spotting in unconstrained hand-written documents. In: ICFHR (2008)

  4. Likforman-Sulem, L., Zahour, A., Taconet, B.: Text line segmentation of historical documents: a survey. Int. J. Doc. Anal. Recognit. 9, 123 (2007)

    Article  Google Scholar 

  5. Louloudis, G., Gatos, B., Pratikakis, I., Halatsis, C.: Text line and word segmentation of handwritten documents. Pattern Recognit. 42, 3169 (2009)

    Article  MATH  Google Scholar 

  6. Leydier, Y., Ouji, A., Lebourgeois, F., Emptoz, H.: Towards an omnilingual word retrieval system for ancient manuscripts. Pattern Recognit. 42, 2089–2105 (2009)

    Article  MATH  Google Scholar 

  7. Zhang, X., Tan, C.L.: Segmentation-free keyword spotting for handwritten documents based on heat kernel signature. In: International Conference on Document Analysis and Recognition, pp. 827–831 (2013)

  8. Frinken, V., Fischer, A., Manmatha, R., Bunke, H.: A novel word spotting method based on recurrent neural networks. IEEE Trans. PAMI 34, 211 (2012)

    Article  Google Scholar 

  9. Rusinol, M., Aldavert, D., Toledo, R., Lladós, J.: Efficient segmentation-free keyword spotting in historical document collections. Pattern Recognit. 48, 545–555 (2014)

    Article  Google Scholar 

  10. Csurka, G., Dance, C.R., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, European Conference on Computer Vision, pp. 1–22 (2004)

  11. Rothacker, L., Rusiñol, M., Fink, G.A.: Bag-of-features HMMs for segmentation-free word spotting in handwritten documents. In: International Conference on Document Analysis and Recognition, pp. 1305–1309 (2013)

  12. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Segmentation-free word spotting with exemplar SVMs. Pattern Recognit. 47, 3967 (2014)

    Article  Google Scholar 

  13. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and recognition with embedded attributes. IEEE Trans. PAMI 36, 2552 (2014)

    Article  Google Scholar 

  14. Sudholt, S., Fink, G.A.: PHOCNet: a deep convolutional neural network for word spotting in handwritten documents. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 277–282 (2016)

  15. Poznanski, A., Wolf, L.: CNN-N-gram for handwriting word recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2305–2314 (2016)

  16. Ghosh, S.K., Valveny, E.: A sliding window framework for word spotting based on word attributes. In: Iberian Conference on Pattern Recognition and Image Analysis, pp. 652–661 (2015)

  17. Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from Edges. In: ECCV (2014)

  18. Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. Int. J. Comput. Vis. 104, 154 (2013)

    Article  Google Scholar 

  19. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. IJCV 116, 1 (2015)

    Article  MathSciNet  Google Scholar 

  20. Kovalchuk, A., Wolf, L, Dershowitz, N.: A simple and fast word spotting method. In: International Conference on Frontiers in Handwriting Recognition (2014)

  21. Wilkinson, T., Brun, A.: A novel word segmentation method based on object detection and deep learning. In: International Symposium of Advances in Visual Computing, pp. 231–240 (2015)

  22. Ghosh, S.K., Valveny, E.: Query by string word spotting based on character bi-gram indexing. In: 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 881–885 (2015)

  23. Santosh, K.C.: g-DICE: graph mining-based document information content exploitation. Int. J. Doc. Anal. Recognit. (IJDAR) 18, 337–355 (2015)

    Article  Google Scholar 

  24. Hassan, Tamir: User-guided wrapping of pdf documents using graph matching techniques. In: International Conference on Document Analysis and Recognition, pp. 631–635 (2009)

  25. Leslie, C., Eskin, E., Noble, W.: The spectrum kernel: a string kernel for SVM protein classification. In: Pacific Symposium on Biocomputing (2002)

  26. Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., Watkins, C.: Text classification using string kernels. JMLR 2, 419 (2002)

    MATH  Google Scholar 

  27. Perronnin, F., Sánchez, J., Mensink, T.: Improving the Fisher kernel for large-scale image classification. In: ECCV (2010)

  28. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE CVPR (2005)

  29. Jones, M.N., Mewor, D.J.K.: Case-sensitive letter and bigram frequency counts from large-scale English corpora. Behav Res Methods Instrum Comput 36, 388 (2004)

    Article  Google Scholar 

  30. Rath, T., Manmatha, R.: Word spotting for historical documents. IJDAR 9, 139 (2007)

    Article  Google Scholar 

  31. Marti, U.-V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. IJDAR 5, 39 (2002)

    Article  MATH  Google Scholar 

  32. Romero, V., Fornés, A., Serrano, N., Sánchez, J.A., Toselli, A.H., Frinken, V., Vidal, E., Lladós, J.: The ESPOSALLES Database: an ancient marriage license corpus for off-line handwriting recognition. Pattern Recognit. (PR) 46, 1658 (2013)

    Article  Google Scholar 

  33. Leydier, Y., Lebourgeois, F., Emptoz, H.: Text search for medieval manuscript images. Pattern Recognit. 40(12), 3552 (2007)

    Article  MATH  Google Scholar 

  34. Terasawa, K., Tanaka, Y.: Slit style HOG feature for document image word spotting. In: Proceedings of the International Conference on Document Analysis and Recognition (2009)

  35. Wilkinson, T., Lindström, J., Brun, A.: Neural Ctrl-F: segmentation-free query-by-string word spotting in handwritten manuscript collections. In: IEEE Conference on Computer Vision (2017)

  36. Rothacker, L., Fink, G.A.: Segmentation-free query-by-string word spotting with bag-of-features HMMs. In: 13th International Conference on Document Analysis and Recognition (ICDAR) (2015)

  37. Liang, Y., Fairhurst, M., Guest, R.: A synthesised word approach to word retrieval in handwritten documents. Pattern Recognit. 45, 4225 (2012)

    Article  Google Scholar 

  38. Fischer, A., Keller, A., Frinken, V., Bunke, H.: Lexicon-free handwritten word spotting using character HMMs. Pattern Recognit. Lett. 33, 934 (2012)

    Article  Google Scholar 

  39. Krishnan, P., Jawahar, C.V.: Matching handwritten document images. In: ECCV, pp. 766–782 (2016)

Download references

Author information

Authors and Affiliations

  1. Computer Vision Center, Dept. Ciències de la Computació, Universitat Autònoma de Barcelona, 08193, Bellaterra (Barcelona), Spain

    Suman Ghosh & Ernest Valveny

Authors
  1. Suman Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ernest Valveny
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suman Ghosh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghosh, S., Valveny, E. Text box proposals for handwritten word spotting from documents. IJDAR 21, 91–108 (2018). https://doi.org/10.1007/s10032-018-0300-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-018-0300-7

Keywords