Skip to main content

Advertisement

Springer Nature Link
Log in

Domain and writer adaptation of offline Arabic handwriting recognition using deep neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Arabic Handwritten Text Recognition (AHTR) based on deep learning approaches remains a challenging problem due to the inevitable domain shift like the variability among writers’ styles and the scarcity of labelled data. To alleviate such problems, we investigate in this paper different domain adaptation strategies of AHTR system. The main idea is to exploit the knowledge of a handwriting source domain and to transfer this knowledge to another domain where only few labelled data are available. Different writer-dependent and writer-independent domain adaptation strategies are explored using a convolutional neural networks (CNN) and Bidirectional Long Short Term Memory (BSTM) - connectionist temporal classification (CTC) architecture. To discuss the interest of the proposed techniques on the target domain, we have conducted extensive experiments using three Arabic handwritten text datasets, mainly, the MADCAT, the AHTID/MW and the IFN/ENIT. Concurrently, the Arabic handwritten text dataset KHATT was used as the source domain. The obtained results prove the effectiveness of the proposed strategies specially when considering the writer’s information during the supervised adaptation process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Kessentini Y, Paquet T, Hamadou A B (2007) A multi-stream approach to off-line handwritten word recognition, in: international Conference on Document Analysis and Recognition, ICDAR, Vol. 7, p. 317-321

  2. Bernard AB, Menasri F, El-Hajj R, Mokbel C, Kermorvant C, Likforman L (2011) Dynamic and contextual information in HMM Behaviour for handwritten word recognition. IEEE Trans Pattern Anal Mach Intell 99:2066–2080

    Article  Google Scholar 

  3. Koerich AL, Sabourin R, Suen CY (2003) Lexicon-driven HMM decoding for large vocabulary handwriting recognition with multiple character models. Int J Doc Anal Recognit, IJDAR 6:126–144

    Article  Google Scholar 

  4. Graves A, Eck D, Beringer N, Schmidhuber J, Biologically plausible speech recognition with LSTM neural nets, in: Biologically Inspired Approaches to Advanced Information Technology, 2019, p. 127-136

  5. Graves A, Liwicki M, Fernández S, Bertolami R, Bunke H, Schmidhuber J (2009) A novel connectionist system for unconstrained handwriting recognition. IEEE Trans Pattern Anal Mach Intell 31:855–68

    Article  Google Scholar 

  6. Wollmer M, Metallinou A, Eyben F, Schuller B, Narayanan S S, (2010) Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling, in: Biologically plausible speech recognition with LSTM neural nets, p. 2362-2365

  7. Gers F A, Eck D, Schmidhuber J, (2001) Applying LSTM to Time Series Predictable Through Time-Window Approaches. Artificial Neural Networks, in: Artificial Neural Networks, ICANN, p. 669-676

  8. Jaeger H (2002) Tutorial on Training Recurrent Neural Networks, Covering BPTT, RTRL, EKF and the Echo State Network’ approach. Sankt Augustin. Tech Rep, Germany

    Google Scholar 

  9. Graves A, (2008) Supervised sequence labelling with recurrent neural networks. Ph.D. dissertation, Ph.D. dissertation, Technical University Munich,

  10. Graves A, S Fernández, Gomez F, Schmidhuber J, (2006) Connectionist Temporal Classification : labelling unsegmented sequence data with recurrent neural networks, in: International conference on Machine learning, ICML, p. 369-376

  11. Jemni S K, Kessentini Y, Kanoun S, Ogier J, (2018) Offline Arabic Handwriting Recognition Using BLSTMs Combination, in: IAPR International Workshop on Document Analysis Systems, DAS, p. 31-36

  12. Cherawala Y, Roy PP, Cheriet M (2017) Combination of context-dependent bidirectional long short-term memory classifiers for robust offline handwriting recognition. Pattern Recognit Lett 90:58–64

    Article  Google Scholar 

  13. Oprean C, Likforman-Sulem L, Popescu A, Mokbel C, (2015) BLSTM-based handwritten text recognition using Web resources, in: International Conference on Document Analysis and Recognition, ICDAR, p. 466-470

  14. Jemni SK, Kessentini Y, Kanoun S (2020) Improving recurrent neural networks for offline arabic handwriting recognition by combining different language models. Int J Pattern Recognit Artif Intell. https://doi.org/10.1142/S0218001420520072

    Article  Google Scholar 

  15. Jemni SK, Kessentini Y, Kanoun S (2019) Out of vocabulary word detection and recovery in Arabic handwritten text recognition. Pattern Recognit 93:507–520

    Article  Google Scholar 

  16. Graves A, Schmidhuber J (2008) Offline handwriting recognition with multidimensional recurrent neural networks. Adv Neural Inform Process Syst 21:545–552

    Google Scholar 

  17. Louradour J, Kermorvant C, (2013) Curriculum Learning for Handwritten Text Line Recognition, arxiv preprint arxiv :1312.1737 1-9

  18. Chherawala Y, Roy PP, Cheriet M (2016) Feature set evaluation for offline handwriting recognition systems: application to the recurrent neural network model. IEEE Trans Cybern 46:2825–2836

    Article  Google Scholar 

  19. Castro D, Bezerra B L D, Valenca M, (2018) Boosting the deep multidimensional long-short-term memory network for handwritten recognition systems, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, p. 127-132

  20. Simard P Y, Steinkraus D, Platt J C, (2003) Best practices for convolutional neural net- works applied to visual document analysis, in: International Conference on Document Analysis and Recognition, ICDAR, p. 958-962

  21. Eltay M, Zidouri A, Ahmad I (2020) Exploring deep learning approaches to recognize handwritten Arabic texts. IEEE Access 8:89882–89898

    Article  Google Scholar 

  22. Voigtlaender P, Doetsch P, Ney H, (2016) Handwriting Recognition with Large Multidimensional Long Short-Term Memory Recurrent Neural Networks, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, p. 228-233

  23. Altwaijry N, Al-Turaiki I (2021) Arabic handwriting recognition system using convolutional neural network neural computing and applications. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05070-8

    Article  Google Scholar 

  24. Bahdanau D, Cho K, Bengio Y, (2014) Neural machine translation by jointly learning to align and translate, in: arXiv:1409.0473, p

  25. Li J, Jin K, Zhou D, Kubota L, Ju Z (2020) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411:340–350

    Article  Google Scholar 

  26. Bluche T, Louradour J, Messina R (2017) Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention, in: International Conference on Document Analysis and Recognition, ICDAR, p. 1050-1055

  27. Michael J, Labahn R, Gruning T, Zollner J, (2019) Evaluating Sequence - to - Sequence Models for Handwritten Text Recognition, in: International Conference on Document Analysis and Recognition, ICDAR, p. 1286-1293

  28. Le A D, Nguyen H T, Nakagawa M, (2020) End to End Recognition System for Recognizing Offline Unconstrained Vietnamese Handwriting, in: SN Computer Science, Vol. 7, pp. 1–8

  29. T. Bluche, R. Messina, (2017) Gated Convolutional Recurrent Neural Networks for Multilingual Handwriting Recognition, in: Proceeding of International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp. 646–651

  30. Poulos J, Valle R (2021) Character-based handwritten text transcription with attention networks. Neural Comput Appl. https://doi.org/10.1007/s00521-021-05813-1

    Article  Google Scholar 

  31. Shimodaira H (2000) Improving predictive inference under covariate shift by weighting the log-likelihood function. J Statist Plan Inference 90(2):227–244. https://doi.org/10.1142/S0218001420520072

    Article  MathSciNet  MATH  Google Scholar 

  32. Ponce J, Berg T L, Everingham M, Forsyth D A, Hebert M, Lazebnik S, Marszalek M, Schmid C, Russell B C, Torralba A, et al. (2006) J, Dataset issues in object recognition, in: Toward category-level object recognition, p. 29-48

  33. Long M, Cao Y, Wang J, Jordan M I, (2015) Learning transferable features with deep adaptation networks, in: arXiv preprint arXiv:1502.02791 p

  34. Motiian S, Jones Q, Iranmaesh SM, Doretto G, (2017) Few-Shot Adversarial Domain Adaptation, in: Conference on Neural Information Processing Systems, NIPS, p

  35. Fernando B, Habrard A, Sebban M, Tuytelaars T, (2013)Unsupervised visual domain adaptation using subspace alignment, in: IEEE ICCV, p. 2960-2967

  36. Fernando B, Habrard A, Sebban M, Tuytelaars T, (2016) Learning the roots of visual domain shift, in: Computer Vision-ECCV 2016 Workshops, p. 475-482

  37. Zhang XY, Liu CL (2013) Writer adaptation with style transfer mapping. IEEE Trans Pattern Anal Mach Intel 35(7):1773–1787

    Article  Google Scholar 

  38. Ahmad I, Fink G A, (2015) Training an arabic handwriting recognizer without a handwritten training dataset, in: nternational Conference on Document Analysis and Recognition, ICDAR, p. 476-480

  39. Serrano JAR, Perronnin F, Sanchez G, Llados J (2010) Unsupervised writer adaptation of whole-word HMMs with application to word-spotting. Pattern Recogn Lett 31(8):742–748

    Article  Google Scholar 

  40. Nosary A, Heutte L, Paquet T (2004) Unsupervised writer adaptation applied to handwritten text recognition. Pattern Recognit 37(2):385–388

    Article  Google Scholar 

  41. Aradillas J C, Murillo-Fuentes J J, Olmos P M, (2018)Boosting Handwriting Text Recognition in Small Databases with Transfer Learning, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, no. 429-434, https://doi.org/10.1109/ICFHR-2018.2018.00081

  42. Lei K, Marçal R, Alicia F, Pau R, Mauricio V, (2020) Unsupervised adaptation for syntheticto-real handwritten word recognition, in: WACV,

  43. Souibgui M A, Fornés A, Kessentini Y, Tudor C, (2021) A Few-shot Learning Approach for Historical Ciphered Manuscript Recognition, in: International Conference on Pattern Recognition, ICPR, pp. 5413–5420

  44. Granet A, Morin E, Mouchere H, Quiniou S, Gaudin C V, (2018) Transfer learning for handwriting recognition on historical documents, in: International Conference on Pattern Recognition Applications and Methods, ICPRAM, p. 432-439

  45. Yang H M, Zhang X Y, Yin F, Sun J, Liu C L, (2018) Deep transfer mapping for unsupervised writer adaptation, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, p. 151-156

  46. Kang L, Rusinol M, Fornés A, Riba P, Villegas M, (2020) Unsupervised Adaptation for Synthetic-to-Real Handwritten Word Recognition, in: IEEE Winter Conference on Applications of Computer Vision, WACV, p. 3491-3500

  47. Puigcerver J, (2017) Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?, in: International Conference on Document Analysis and Recognition, ICDAR, p. 67-72

  48. Miseikis J, Brijačak I, Yahyanejad S, Glette K, Elle O J, Torresen J, (2019) Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image Using CNN, in: International Conference on Robotics and Automation, ICRA, p

  49. Oquab M, Bottou L, Laptev I, Sivic J, (2014) Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks, in: IEEE Conference on Computer Vision and Pattern Recognition, p. 1717-1724

  50. Graves A, Fernandez S, Gomez F, Schmidhuber J, (2006) Connectionist temporal classifcation: Labelling unsegmented sequence data with recurrent neural’ networks, in: ICM of the International Conference on Machine Learning, p. 369-376

  51. Mahmoud S A, Ahmad I, Alshayeb M, Al-Khatib W G, Parvez M T, Fink G A, Margner V, El Abed H, (2012) KHATT: Arabic offline handwritten text database, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, no. 449-454,

  52. Strassel S, (2009) Linguistic resources for Arabic handwriting recognition, in: International Conference on Arabic Language Resources and Tools, no. 37-41

  53. Mezghani A, Kanoun S, Khemakhem M, (2012) A Database for Arabic Handwritten Text Image Recognition and Writer Identification, in: International Conference on Frontiers in Handwriting Recognition, ICFHR, no. 399-402

  54. Pechwitz M, Maddouri S S, Margner V, Ellouze N, Amiri H, (2002) IFN/ENIT-database of handwritten Arabic words, in: Colloque International Francophone sur l’Ecrit et le Document, CIFED, no. 129-136,

  55. V. I. Levenshtein, Binary codes capable of correcting deletions, insertions, and reversals, in: Soviet physics doklady, Vol. 10, Soviet Union, 1966, pp. 707–710

  56. Kingma D P, Ba J, (2015) Adam: A Method for Stochastic Optimization, in: International Conference for Learning Representations, p

  57. Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7):1895–1923. https://doi.org/10.1162/089976698300017197

    Article  Google Scholar 

  58. Mohri M, Pereira F, Riley M (2002) Weighted finite-state transducers in speech recognition. Comput Speech Lang 16(1):69–88. https://doi.org/10.1006/csla.2001.0184

    Article  Google Scholar 

  59. M. Pechwitz, V. Maergner, (2003) Hmm based approach for handwritten arabic word recognition using the ifn/enit - database, in: International Conference on Document Analysis and Recognition, pp. 890–894. https://doi.org/10.1109/ICDAR.2003.1227788

  60. Al-Hajj R, Mokbel C, Likforman-Sulem L, (2007) Combination of HMM-based classifers for the recognition of Arabic handwritten words, in: Proceeding of International Conference on Document Analysis and Recognition (ICDAR), pp. 959–963

  61. P. Dreuw, S. Jonas, H. Ney, (2008) White-space models for offline arabic handwriting recognition, in: 2008 19th International Conference on Pattern Recognition, pp. 1–4. https://doi.org/10.1109/ICPR.2008.4761841

  62. P. Dreuw, D. Rybach, C. Gollan, H. Ney, (20090) Writer adaptive training and writing variant model refinement for offline arabic handwriting recognition, IEEE Computer Society, USA. https://doi.org/10.1109/ICDAR.2009.9

  63. Kessentini Y, Paquet T, Ben Hamadou A (2010) Off-line handwritten word recognition using multistream hidden Markov models. Pattern Recognit Lett 31:60–70

    Article  Google Scholar 

  64. AlKhateeb JH, Ren J, Jiang J, Al-Muhtaseb H (2011) Offline handwritten arabic cursive text recognition using hidden markov models and re-ranking. Pattern Recognit Lett 32(8):1081–1088. https://doi.org/10.1016/j.patrec.2011.02.006

    Article  Google Scholar 

  65. Azeem SA, Ahmed H (2013) Effective technique for the recognition of offline arabic handwritten words using hidden markov models. Int J Doc Anal Recognit, IJDAR 16(8):399–412. https://doi.org/10.1109/ACCESS.2020.2994248

    Article  Google Scholar 

  66. Jayech K, Mahjoub M, Amara N (2016) Arabic handwritten word recognition based on dynamic bayesian network. Int Arab J Inf Technol 13:1024–1031

    Google Scholar 

  67. Rabi M, Amrouch M, Mahani Z (2018) Recognition of cursive arabic handwritten text using embedded training based on hidden markov models. Int J Pattern Recognit Artif Intell 32(01):1860007. https://doi.org/10.1142/S0218001418600078

    Article  Google Scholar 

  68. Amrouch M., Rabi M., Es-Saady Y., (2018) Convolutional Feature Learning and CNN Based HMM for Arabic Handwriting Recognition, in: Image and Signal Processing, ICISP, Lecture Notes in Computer Science, Vol. 10884, pp. 5413–5420. https://doi.org/10.1007/978-3-319-94211-7_29

  69. Tavoli R, Keyvanpour M, Mozaffari S (2018) Statistical geometric components of straight lines (sgcsl) feature extraction method for offline arabic/persian handwritten words recognition. IET Image Process 12(9):1606–1616

    Article  Google Scholar 

  70. T. M. Ghanim, M. Khalil, H. M. Abbas, (2019) Multi-stage off-line arabic handwriting recognition approach using advanced cascading technique, in: ICPRAM,

  71. Ahmad I, Fink GA (2019) Handwritten arabic text recognition using multi-stage sub-core-shape hmms. Int J Doc Anal Recognit 22:329–349. https://doi.org/10.1007/s10032-019-00339-8

    Article  Google Scholar 

  72. Sulaiman A, Omar K, Nasrudin MF (2021) Two streams deep neural network for handwriting word recognition. Multim Tools Appl 80(8):5473–5494. https://doi.org/10.1007/s11042-020-09923-1

    Article  Google Scholar 

  73. Bhunia AK, Chowdhury PN,  Yang Y,  Hospedales T, Xiang T, Song YZ (2021) Vectorization and rasterization: Self-supervised learning for sketch and handwriting, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Download references

Author information

Authors and Affiliations

  1. Digital Research Center of Sfax, B.P. 275, Sakiet Ezzit, 3021 Sfax, Tunisia

    Sana Khamekhem Jemni, Sourour Ammar & Yousri Kessentini

  2. MIR@CL: Multimedia, InfoRmation systems and Advanced Computing Laboratory, Sfax, Tunisia

    Sana Khamekhem Jemni

  3. SM@RTS : Laboratory of Signals, systeMs, aRtificial Intelligence and neTworkS, Sfax, Tunisia

    Sourour Ammar & Yousri Kessentini

Authors
  1. Sana Khamekhem Jemni
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Sourour Ammar
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yousri Kessentini
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Sana Khamekhem Jemni.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jemni, S.K., Ammar, S. & Kessentini, Y. Domain and writer adaptation of offline Arabic handwriting recognition using deep neural networks. Neural Comput & Applic 34, 2055–2071 (2022). https://doi.org/10.1007/s00521-021-06520-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06520-7

Keywords

Profiles

  1. Sourour Ammar View author profile