Skip to main content
SpringerLink
Log in

Multimodal Scoring Model for Handwritten Chinese Essay

  • Conference paper
  • First Online:
Document Analysis and Recognition - ICDAR 2023 (ICDAR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14187))

Included in the following conference series:

  • 1144 Accesses

Abstract

Essay writing plays a critical role in Chinese language skill teaching. With the smart education becomes a hot topic, the demand for automatic essay scoring (AES) has been emerging among teachers and students. Existing works frequently ignore the impact of the visual modal during the scoring process, such as writing quality in terms of neatness or legibility. This paper addresses the problem with a visual-textual integrating perspective and proposes a deep learning based multi-modal AES. Specifically, implicit alignment algorithm is presented to cohere the distinct visual modal and text modal. Methods are tested on a large-scale dataset consisting of over 4000 essays including HSK publicly available samples. The results show that multi-modal AES reduce the MAE of scoring from 1.13 to 1.06, and the implicit alignment algorithm reduces it further to 1.01.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. BLCU: HSK dynamic essay corpus ver 2.0 (in Chinese). https://hsk.blcu.edu.cn/. (26 Apr 2022)

  2. Che, W., Feng, Y., Qin, L., Liu, T.: N-LTP: An open-source neural language technology platform for Chinese. In: Proceedings of NAACL-SD, pp. 42–49 (2021)

    Google Scholar 

  3. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  4. He, J., Zhang, C., Li, X., Zhang, D.: Survey of research on multimodal fusion technology for deep learning (in Chinese). Comput. Eng. 46(5), 1–11 (2020)

    Google Scholar 

  5. Hu, R., Xiao, H.: The construction of Chinese collocation knowledge bases and their application in second language acquisition. In: Applied Linguistics. vol. 1 (2019)

    Google Scholar 

  6. Huang, Z., Xie, J., Xun, E., et al.: Study of feature selection in HSK automated essay scoring. Comput. Eng. Appl. 6, 118–122 (2014)

    Google Scholar 

  7. Huang, Z., Xie, J., Xun, E.: Study of feature selection in HSK automated essay scoring (in Chinese). Comput. Eng. Appl. 6, 118–122 (2014)

    Google Scholar 

  8. Laufer, B., Nation, P.: Vocabulary size and use: lexical richness in L2 written production. Appl. Linguist. 16(3), 307–322 (1995)

    Article  Google Scholar 

  9. Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D.: Unicoder-VL: a universal encoder for vision and language by cross-modal pre-training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 11336–11344 (2020)

    Google Scholar 

  10. Li, H., Dai, T.: Explore deep learning for Chinese essay automated scoring. In: Journal of Physics: Conference Series. vol. 1631, p. 012036. IOP Publishing (2020)

    Google Scholar 

  11. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)

  12. Lonsdale, D., Strong-Krause, D.: Automated rating of ESL essays. In: Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications using Natural Language Processing, pp. 61–67 (2003)

    Google Scholar 

  13. Nadeem, F., Nguyen, H., Liu, Y., Ostendorf, M.: Automated essay scoring with discourse-aware neural models. In: Proceedings of the Fourteenth Workshop on Innovative use of NLP for Building Educational Applications, pp. 484–493 (2019)

    Google Scholar 

  14. Page, E.B.: The use of the computer in analyzing student essays. Int. Rev. Educ. 14, 210–225 (1968)

    Google Scholar 

  15. Su, T., You, H., Liu, S., Wang, Z.: FPRNet: end-to-end full-page recognition model for handwritten Chinese essay. In: Porwal, U., Fornés, A., Shafait, F. (eds) Frontiers in Handwriting Recognition. ICFHR 2022. Lecture Notes in Computer Science. vol 13639. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-21648-0_16

  16. Su, W., et al.: VL-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019)

  17. Tan, H., Bansal, M.: LXMERT: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 (2019)

  18. Tapaswi, M., Bäuml, M., Stiefelhagen, R.: Aligning plot synopses to videos for story-based retrieval. Int. J. Multimedia Inf. Retrieval 4(1), 3–16 (2015)

    Article  Google Scholar 

  19. Tay, Y., Phan, M., Tuan, L.A., Hui, S.C.: SkipFlow: incorporating neural coherence features for end-to-end automatic text scoring. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)

    Google Scholar 

  20. Van Hout, R., Vermeer, A.: Comparing measures of lexical richness. Model. Assessing Vocabulary Knowl. 93, 115 (2007)

    Google Scholar 

  21. Wang, Y., Hu, R.: A prompt-independent and interpretable automated essay scoring method for Chinese second language writing. In: Li, S. (ed.) CCL 2021. LNCS (LNAI), vol. 12869, pp. 450–470. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-84186-7_30

    Chapter  Google Scholar 

  22. Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192–1200 (2020)

    Google Scholar 

  23. Zhang, P., et al.: VSR: a unified framework for document layout analysis combining vision, semantics and relations. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12821, pp. 115–130. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86549-8_8

    Chapter  Google Scholar 

Download references

Acknowledgment

This work was supported by the National Key Research and Development Program of China (Grant No. 2020AAA0108003) and National Natural Science Foundation of China (Grant No. 62277011 and 61673140).

Author information

Authors and Affiliations

  1. School of Software, Harbin Institute of Technology, Harbin, People’s Republic of China

    Tonghua Su, Jifeng Wang, Hongming You & Zhongjie Wang

Authors
  1. Tonghua Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jifeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongming You
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhongjie Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tonghua Su .

Editor information

Editors and Affiliations

  1. TU Dortmund University, Dortmund, Germany

    Gernot A. Fink

  2. Adobe, College Park, MN, USA

    Rajiv Jain

  3. Osaka Metropolitan University, Osaka, Japan

    Koichi Kise

  4. Rochester Institute of Technology, Rochester, NY, USA

    Richard Zanibbi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Su, T., Wang, J., You, H., Wang, Z. (2023). Multimodal Scoring Model for Handwritten Chinese Essay. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14187. Springer, Cham. https://doi.org/10.1007/978-3-031-41676-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41676-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41675-0

  • Online ISBN: 978-3-031-41676-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation