Skip to main content
SpringerLink
Log in

Handwritten Text Segmentation via End-to-End Learning of Convolutional Neural Networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We present a method that separates handwritten and machine-printed components that are mixed and overlapped in documents. Many conventional methods addressed this problem by extracting connected components (CCs) and classifying the extracted CCs into two classes. They were based on the assumption that two types of components are not overlapping each other, while we are focusing on more challenging and realistic cases where the components are often overlapping each other. For this, we propose a new method that performs pixel-level classification with a convolutional neural network. Unlike conventional neural network methods, our method works in an end-to-end manner and does not require any preprocessing steps (e.g., foreground extraction, handcrafted feature extraction, and so on). For the training of our network, we develop a cross-entropy based loss function to alleviate the class imbalance problem. Regarding the training dataset, although there are some datasets of mixed printed characters and handwritten scripts, most of them do not have overlapping cases and do not provide pixel-level annotations. Hence, we also propose a data synthesis method that generates realistic pixel-level training samples having many overlappings of printed and handwritten components. Experimental results on synthetic and real images have shown the effectiveness of the proposed method. Although the proposed network has been trained only with synthetic images, it also improves the OCR rate of real documents. Specifically, the OCR rate for machine-printed texts is increased from 0.8087 to 0.9442 by removing the overlapped handwritten scribbles by our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Antonacopoulos A, Bridson D, Papadopoulos C, Pletschacher S (2009) “A realistic dataset for performance evaluation of document layout analysis”. In: Document Analysis and Recognition, 2009. ICDAR’09. 10th International Conference on IEEE, pp 296–300

  2. Bhowmik S, Kundu S, De BK, Sarkar R, Nasipuri M (2019) A Two-Stage Approach for Text and Non-text Separation from Handwritten Scientific Document Images. In: Information Technology and Applied Mathematics, Springer, Singapore, pp 41–51

  3. Breuel TM (2003) “High performance document layout analysis.” Proceedings of the Symposium on Document Image Understanding Technology

  4. Franke J, Oberlander M (1993) “Writing style detection by statistical combination of classifiers in form reader applications”. In: Document Analysis and Recognition, 1993., Proceedings of the Second International Conference on. IEEE, pp. 581–584

  5. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  6. Kandan R, Reddy NK, Arvind KR, Ramakrishnan AG (2007) “A robust two level classification algorithm for text localization in documents”. In: International Symposium on Visual Computing Springer, pp 96–105

  7. Kingma DP, Ba J (2014) “Adam: A method for stochastic optimization”. arXiv preprint arXiv:1412.6980

  8. Koo HI, Cho NI (2012) Text-line extraction in handwritten Chinese documents based on an energy minimization framework. IEEE Trans Image Process 21(3):1169–1175

    Article  MathSciNet  Google Scholar 

  9. Levenshtein VI (1966) “Binary codes capable of correcting deletions, insertions, and reversals”. Soviet physics doklady 8:10

    Google Scholar 

  10. Li X-H, Yin F, Liu C-L (2018) “Printed/handwritten texts and graphics separation in complex documents using conditional random fields”. In: 2018 13th IAPR International Workshop on Document Analysis Systems (DAS) IEEE, pp 145–150

  11. Lin T-Y, Goyal P, Girshick R, He K, Doll’ar P (2018) “Focal loss for dense object detection,” IEEE transactions on pattern analysis and machine intelligence

  12. Lin Y, Song Y, Li Y, Wang F, He K (2017) Multilingual corpus construction based on printed and handwritten character separation. Multimedia Tools and Applications 76(3):4123–4139

    Article  Google Scholar 

  13. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  14. Marti U-V, Bunke Horst (2002) The iam-database: an english sentence database for offline handwriting recognition. Int J Doc Anal Recognit 5(1):39–46

    Article  Google Scholar 

  15. Neelima KB, Arulselvi S (2020) Classification of printed text and handwritten characters with neural networks. Journal of Critical Reviews 7(2):134–139

    Google Scholar 

  16. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics 9(1):62–66

    Article  Google Scholar 

  17. Peng X, Setlur S, Govindaraju V, Sitaram R (2013) Handwritten text separation from annotated machine printed documents using markov random fields. International Journal on Document Analysis and Recognition (IJDAR) 16 (1):1–16

    Article  Google Scholar 

  18. Ronneberger O, Fischer P, Brox T (2015) “u-net: Convolutional networks for biomedical image segmentation”. In: International Conference on Medical image computing and computer-assisted intervention Springer, pp 234–241

  19. Ryu J, Koo HI, Cho NI (2014) Language-independent text-line extraction algorithm for handwritten documents. IEEE Signal processing letters 21 (9):1115–1119

    Article  Google Scholar 

  20. Sahare P, Dhok SB (2019) Separation of Machine-Printed and Handwritten Texts in Noisy Documents using Wavelet Transform, vol 36, pp 341–361

  21. Seuret M, Liwicki M, Ingold R (2014) “Pixel level handwritten and printed content discrimination in scanned documents”. In: Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on IEEE, pp 423–428

  22. Smith R (2007) “An overview of the tesseract ocr engine”. In: Document Analysis and Recognition. ICDAR 2007. Ninth International Conference on. IEEE, 2007, vol. 2, pp. 629–633

  23. Zhou Z-H, Liu X-Y (2006) Training costsensitive neural networks with methods addressing the class imbalance problem. IEEE Trans Knowl Data Eng 18 (1):63–77

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.NI190004,Development of AI based Robot Technologies for Understanding Assembly Instruction and Automatic Assembly Task Planning), and in part by Hancom Inc.

Author information

Authors and Affiliations

  1. Dept. of Electrical and Computer Engineering, Seoul National University, Gwanak-ro 1, Gwanak-Gu, Seoul, 08826, Korea

    Nam Ik Cho

  2. Dept. of Electrical and Computer Eng., INMC, Seoul National University, Seoul, Korea

    Junho Jo, Jae Woong Soh & Nam Ik Cho

  3. Dept. of Electrical and Computer Engineering, Ajou University, Suwon, Korea

    Hyung Il Koo

Authors
  1. Junho Jo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyung Il Koo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jae Woong Soh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nam Ik Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nam Ik Cho.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jo, J., Koo, H.I., Soh, J.W. et al. Handwritten Text Segmentation via End-to-End Learning of Convolutional Neural Networks. Multimed Tools Appl 79, 32137–32150 (2020). https://doi.org/10.1007/s11042-020-09624-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09624-9

Keywords

Search

Navigation