Skip to main content
SpringerLink
Log in

Crosslingual Handwritten Text Generation Using GANs

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2023 Workshops (ICDAR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14194))

Included in the following conference series:

Abstract

Generative Adversarial Networks, also called GANs, have been able to produce realistic looking handwritten images. However, training a GAN to generate usable data for improving handwriting recognition is a chicken and egg problem. In a low-resource language scenario, it would be beneficial to have a method that can generate more labeled data. But training such a GAN requires an amount of data that would not be available in a low-resource setting.

In this paper, we present our work in data augmentation with a GAN that is independent of language and can be used to generate handwritten images by learning a mapping between printed and handwritten text. Our method is able to leverage training data from a source language and generate handwriting in a different target language. We show that in scenarios with adequate amounts of target language data, similar improvements in WER can be made by augmenting with either synthetic handwritten or printed text. However, in low resource scenarios, our GAN generated handwriting improves recognition results by 5–10% absolute over the baseline and 3–5% absolute over adding rendered printed text.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/VinhLoiIT/vnondb_convert.

  2. 2.

    https://openslr.org/48.

References

  1. Alonso, E., Moysset, B., Messina, R.: Adversarial generation of handwritten text images conditioned on sequences. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 481–486. IEEE Computer Society, Los Alamitos (2019). https://doi.org/10.1109/ICDAR.2019.00083, https://doi.ieeecomputersociety.org/10.1109/ICDAR.2019.00083

  2. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)

    Google Scholar 

  3. Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=B1xsqj09Fm

  4. Carbune, V., et al.: Fast multi-language LSTM-based online handwriting recognition. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 89–102 (2020). https://doi.org/10.1007/s10032-020-00350-4

    Article  Google Scholar 

  5. CFT SL: Cyrillic handwriting dataset (2022). https://www.kaggle.com/datasets/constantinwerner/cyrillic-handwriting-dataset

  6. Etter, D., Rawls, S., Carpenter, C., Sell, G.: A synthetic recipe for OCR. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 864–869 (2019). https://doi.org/10.1109/ICDAR.2019.00143

  7. Fogel, S., Averbuch-Elor, H., Cohen, S., Mazor, S., Litman, R.: ScrabbleGan: semi-supervised varying length handwritten text generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  8. Goodfellow, I., et al.: Generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 3 (2014). https://doi.org/10.1145/3422622

  9. Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets. In: ICML ’06: Proceedings of the International Conference on Machine Learning (2006)

    Google Scholar 

  10. Graves, A., Fernández, S., Schmidhuber, J.: Bidirectional LSTM networks for improved phoneme classification and recognition. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3697, pp. 799–804. Springer, Heidelberg (2005). https://doi.org/10.1007/11550907_126

    Chapter  Google Scholar 

  11. Heil, R., Vats, E., Hast, A.: Strikethrough removal from handwritten words using CycleGANs. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 572–586. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_38

    Chapter  Google Scholar 

  12. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf

  13. Huu, M.K.N., Ho, S.T., Nguyen, V.T., Ngo, T.D.: Multilingual-GAN: a multilingual GAN-based approach for handwritten generation. In: 2021 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), pp. 1–6 (2021). https://doi.org/10.1109/MAPR53640.2021.9585285

  14. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  15. Kong, Y., et al.: Look closer to supervise better: One-shot font generation via component-based discriminator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13482–13491 (2022)

    Google Scholar 

  16. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 1 training set (2012). https://doi.org/10.35111/9bm5-nz55

  17. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 2 training set (2013). https://doi.org/10.35111/044b-ah68

  18. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 3 training set (2013). https://doi.org/10.35111/w1px-d922

  19. Li, C., Taniguchi, Y., Lu, M., Konomi, S.: Few-shot font style transfer between different languages. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 433–442 (2021)

    Google Scholar 

  20. Liu, W., Liu, F., Ding, F., He, Q., Yi, Z.: XMP-FONT: self-supervised cross-modality pre-training for few-shot font generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7905–7914 (2022)

    Google Scholar 

  21. Luo, C., Zhu, Y., Jin, L., Li, Z., Peng, D.: SLOGAN: handwriting style synthesis for arbitrary-length and out-of-vocabulary text. IEEE Trans. Neural Netw. Learn. Syst. (2022)

    Google Scholar 

  22. Marti, U.V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. Int. J. Doc. Anal. Recogn. 5, 39–46 (2002)

    Article  MATH  Google Scholar 

  23. Nayef, N., et al.: ICDAR 2019 robust reading challenge on multi-lingual scene text detection and recognition - RRC-MLT-2019 (2019). https://doi.org/10.48550/ARXIV.1907.00945, https://arxiv.org/abs/1907.00945

  24. Nguyen, H.T., Nguyen, C.T., Nakagawa, M.: ICFHR 2018 - competition on vietnamese online handwritten text recognition using HANDS-VNonDB (VOHTR2018). In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 494–499 (2018). https://doi.org/10.1109/ICFHR-2018.2018.00092

  25. Park, S., Chun, S., Cha, J., Lee, B., Shim, H.: Multiple heads are better than one: few-shot font generation with multiple localized experts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13900–13909 (2021)

    Google Scholar 

  26. Seitzer, M.: PyTorch-FID: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid (2020). version 0.2.1

  27. Sharma, M., Verma, A., Vig, L.: Learning to clean: a GAN perspective. In: Carneiro, G., You, S. (eds.) ACCV 2018. LNCS, vol. 11367, pp. 174–185. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21074-8_14

    Chapter  Google Scholar 

  28. Tang, L., et al.: Few-shot font generation by learning fine-grained local styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7895–7904 (2022)

    Google Scholar 

  29. Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: Coco-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint: arXiv:1601.07140 (2016). http://vision.cornell.edu/se3/wp-content/uploads/2016/01/1601.07140v1.pdf

  30. Wang, Y., et al.: Espresso: a fast end-to-end neural speech recognition toolkit. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 136–143 (2019). https://doi.org/10.1109/ASRU46091.2019.9003968

  31. Wei, H., Liu, K., Zhang, J., Fan, D.: Data augmentation based on CycleGAN for improving woodblock-printing Mongolian words recognition. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 526–537. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_35

    Chapter  Google Scholar 

  32. Wen, C., et al.: Handwritten Chinese font generation with collaborative stroke refinement. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3882–3891 (2021)

    Google Scholar 

  33. Yim, M., Kim, Y., Cho, H.-C., Park, S.: SynthTIGER: synthetic text image GEneratoR towards better text recognition models. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 109–124. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_8

    Chapter  Google Scholar 

  34. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Computer Vision (ICCV), 2017 IEEE International Conference on (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Language and Speech Processing, Johns Hopkins University, Baltimore, USA

    Chun Chieh Chang, Leibny Paola Garcia Perera & Sanjeev Khudanpur

  2. Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, USA

    Chun Chieh Chang, Leibny Paola Garcia Perera & Sanjeev Khudanpur

Authors
  1. Chun Chieh Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leibny Paola Garcia Perera
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanjeev Khudanpur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chun Chieh Chang .

Editor information

Editors and Affiliations

  1. University of La Rochelle, La Rochelle, France

    Mickael Coustaty

  2. Autonomous University of Barcelona, Bellaterra, Spain

    Alicia Fornés

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chang, C.C., Perera, L.P.G., Khudanpur, S. (2023). Crosslingual Handwritten Text Generation Using GANs. In: Coustaty, M., Fornés, A. (eds) Document Analysis and Recognition – ICDAR 2023 Workshops. ICDAR 2023. Lecture Notes in Computer Science, vol 14194. Springer, Cham. https://doi.org/10.1007/978-3-031-41501-2_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41501-2_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41500-5

  • Online ISBN: 978-3-031-41501-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation