Skip to main content
SpringerLink
Log in

Full-page handwriting recognition and automated essay scoring for in-the-wild essays

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Scoring of handwritten essays in school education settings is a time-consuming task. Normalized assessment and prompt feedback enable a student to improve the articulation, comprehension and overall presentation of ideas. In this work, we present a system that can take in input as the images of the essay sheets and outputs the grade/score of the essay. We show a pipelined approach by combining a handwriting recognition model and automated essay scoring. Current handwriting recognition systems show an excellent transcription performance on the existing public domain dataset. These datasets are primarily captured in a constrained manner. The performance and efficacy of these models on unconstrained data are crucial for text understanding. In our work, we adapt an existing full-page handwriting recognition model to the unconstrained handwritten essay dataset. The full page handwriting recognition model is a deep learning model based on CNN and LSTM layers with explicit modules to identify the start of line, line normalization and text line recognition. The unconstrained dataset is from a national essay competition where students upload the essay after scanning the essay. This dataset is wild in nature as the background, margins, text-fonts and the scanning device make it challenging both visually and algorithmically.We have curated a subset of this dataset for all the experiments in this work and intend to make this dataset publicly available. We further analyze the performance on the downstream task of essay scoring using a set of classical handcrafted features and transformer-based contextual embeddings.We have formulated the problem of essay scoring as a regression task. The pre-trained embeddings/handcrafted features, for each essay, are used as representative features for the essay scoring model. Our results show that there is only a slight performance degradation in the essay scoring task due to transcription errors from the handwriting recognition module. We also show analysis with rubric level scores and handcrafted features to develop a subset of features that directly impact the rubric level score on the essay.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://www.kaggle.com/c/asap-aes/

  2. https://github.com/dali92002/DE-GAN

  3. https://github.com/xiaoyu258/DocProj

  4. https://dev.languagetool.org/

References

  1. Attali Y, Burstein J (2006) Automated essay scoring with e-rater®; v. 2. J Technol, Learn Assessment, vol 4(3)

  2. Bojanowski P, Grave E, Joulin A et al (2016) Enriching word vectors with subword information. arXiv:160704606

  3. Chen H, He B (2013) Automated essay scoring by maximizing human-machine agreement. In: Proceedings of the 2013 conference on empirical methods in natural language processing, pp 1741–1752

  4. Chowdhury A, Vig L (2018) An efficient end-to-end neural model for handwritten text recognition. arXiv:180707965

  5. Devlin J, Chang MW, Lee K et al (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:181004805

  6. Graves A (2012) Supervised sequence labelling with recurrent neural networks. 2012, http://booksgooglecom/books

  7. Graves A, Chmidhuber J (2009) Offline handwriting recognition with multidimensional recurrent neural networks. In: Advances in neural information processing systems, pp 545–552

  8. Graves A, Fernández S, Gomez F et al (2006) Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on machine learning. ACM, pp 369–376

  9. Heylighen F, Dewaele J M (2002) Variation in the contextuality of language: an empirical measure. Foundations Sci 7(3):293–340

    Article  Google Scholar 

  10. Jacobson N (2001) A method for normalizing students’ scores when employing multiple graders. ACM SIGCSE Bulletin 33(4):35–38

    Article  Google Scholar 

  11. Joulin A, Grave E, Bojanowski P et al (2016a) Fasttext.zip: compressing text classification models. arXiv:http://arxiv.org/abs/

  12. Joulin A, Grave E, Bojanowski P et al (2016b) Bag of tricks for efficient text classification. arXiv:http://arxiv.org/abs/

  13. Kakkonen T, Myller N, Timonen J et al (2005) Automatic essay grading with probabilistic latent semantic analysis. In: Proceedings of the second workshop on building educational applications using NLP, pp 29–36

  14. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning, pp 1188–1196

  15. Li X, Zhang B, Liao J et al (2019) Document rectification and illumination correction using a patch-based cnn. ACM Trans Graphics (TOG) 38(6):1–11

    Google Scholar 

  16. Liu J, Xu Y, Zhu Y (2019) Automated essay scoring based on two-stage learning. arXiv:190107744

  17. Marti UV, Bunke H (2002) The iam-database: an english sentence database for offline handwriting recognition. Int J Doc Anal Recognit 5(1):39–46

    Article  MATH  Google Scholar 

  18. Mikolov T, Chen K, Corrado G et al (2013a) Efficient estimation of word representations in vector space. arXiv:13013781

  19. Mikolov T, Grave E, Bojanowski P et al (2018) Advances in pre-training distributed word representations. In: Proceedings of the international conference on language resources and evaluation (LREC 2018)

  20. Mikolov T, Sutskever I, Chen K et al (2013b) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119

  21. Moysset B, Kermorvant C, Wolf C (2017) Full-page text recognition: learning where to start and when to stop. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). IEEE, pp 871–876

  22. Pham V, Bluche T, Kermorvant C et al (2014) Dropout improves recurrent neural networks for handwriting recognition. In: Frontiers in handwriting recognition (ICFHR), 2014 14th international conference on. IEEE, pp 285–290

  23. Puigcerver J (2017) Are multidimensional recurrent layers really necessary for handwritten text recognition?. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), IEEE, pp 67–72

  24. Rowtula V, Oota SR, Jawahar C (2019) Towards automated evaluation of handwritten assessments. In: 2019 international conference on document analysis and recognition (ICDAR). IEEE, pp 426–433

  25. Sharma A, Jayagopi DB (2018a) Automated grading of handwritten essays

  26. Sharma A, Jayagopi DB (2018b) Handwritten essay grading on mobiles using mdlstm model and word embeddings. In: Proceedings of the 11th Indian conference on computer vision, graphics and image processing, pp 1–8

  27. Sharma A, Jayagopi DB (2021) Towards efficient unconstrained handwriting recognition using dilated temporal convolution network. Expert Syst Appl 164:114,004

    Article  Google Scholar 

  28. Souibgui MA, Kessentini Y (2020) De-gan: A conditional generative adversarial network for document enhancement. IEEE Trans Pattern Anal Mach Intell

  29. Srihari S, Collins J, Srihari R et al (2006) Automated scoring of handwritten essays based on latent semantic analysis. In: International workshop on document analysis systems. Springer, pp 71–83

  30. Srihari SN, Srihari RK, Babu P et al (2007) On the automatic scoring of handwritten essays. In: IJCAI, pp 2880–2884

  31. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: NIPS

  32. Voigtlaender P, Doetsch P, Ney H (2016) Handwriting recognition with large multidimensional long short-term memory recurrent neural networks. In: Frontiers in handwriting recognition (ICFHR), 2016 15th international conference on. IEEE, pp 228–233

  33. Wigington C, Stewart S, Davis B et al (2017) Data augmentation for recognition of handwritten words and lines using a cnn-lstm network. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). IEEE, pp 639–645

  34. Wigington C, Tensmeyer C, Davis B et al (2018) Start, follow, read: end-to-end full-page handwriting recognition. In: Proceedings of the European conference on computer vision (ECCV), pp 367–383

  35. Yannakoudakis H, Briscoe T (2012) Modeling coherence in esol learner texts. In: Proceedings of the seventh workshop on building educational applications using NLP, pp 33–43

  36. Yousef M, Bishop TE (2020) Origaminet: weakly-supervised, segmentation-free, one-step, full page text recognition by learning to unfold. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14,710–14,719

  37. Zesch T, Wojatzki M, Scholten-Akoun D (2015) Task-independent features for automated essay grading. In: Proceedings of the tenth workshop on innovative use of NLP for building educational applications, pp 224–232

  38. Zhang H, Litman D (2018) Co-attention based neural network for source-dependent essay scoring. In: Proceedings of the thirteenth workshop on innovative Use of NLP for building educational applications. Association for computational linguistics, new Orleans, Louisiana, pp 399–409, https://doi.org/10.18653/v1/W18-0549; https://aclanthology.org/W18-0549

Download references

Funding

This work was supported by Visvesvaraya PhD Scheme, Ministry of Electronics and Information Technology (MeitY), Government of India under grant number MEITY-PHD-2541.

Author information

Authors and Affiliations

  1. Multimodal Perception Lab, International Institute of Information Technology (IIITB), Bangalore, 560100, Karnataka, India

    Annapurna Sharma, Rohit Katlaa, Gurleen Kaur & Dinesh Babu Jayagopi

Authors
  1. Annapurna Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rohit Katlaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gurleen Kaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dinesh Babu Jayagopi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Annapurna Sharma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, A., Katlaa, R., Kaur, G. et al. Full-page handwriting recognition and automated essay scoring for in-the-wild essays. Multimed Tools Appl 82, 35253–35276 (2023). https://doi.org/10.1007/s11042-023-14558-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14558-z

Keywords

Search

Navigation