Abstract
Automated Essay Evaluation (AEE) use a set of features to evaluate and score students essay solutions. Most of the features like lexical similarity, syntax, vocabulary and shallow content were addressed but evaluating students essays using the semantics and context of the essay are not addressed well. To address the issue which are related to the semantics and context, we propose a layered approach to AEE which uses neural word embedding in order to evaluate student answers semantically and the similarity will be computed by using Word Mover’s Distance. We also implemented a plagiarism detection algorithms to protect the students from submitting someone else solution as their own using k-shingles and local sensitive hashing. We also implemented an algorithm that penalize students who are trying to fool the system by submitting only content bearing works. The performance of the proposed AEE was evaluated and compared to other state-of-the-art methods qualitatively and quantitatively. The experimental results show that the proposed AEE approach using neural word embedding achieve higher level of accuracy as compared to others baselines and are promising in evaluating students essay solutions semantically.
Keywords
Tomáš Horváth is also associated with the Institute of Computer Science of the Faculty of Science at the Pavol Jozef Šafárik University in Košice, Slovakia.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Miller, M.D., Linn, R.L., Gronlund, N.E.: Measurement and Assessment in Teaching, 11th edn. Pearson, London (2013)
Page, E.B.: Grading essays by computer: progress report. In: Invitational Conference on Testing Problems (1966)
Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41, 391–407 (1999)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Attali, Y.: A differential word use measure for content analysis in automated essay scoring. ETS Res. Rep. Ser. 36, i–19 (2011)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26, pp. 3111–3119 (2013)
Kusner, M.J., Sun, Y., Kolkin, N.I., Weinberger, K.Q.: From word embeddings to document distances. In: International Conference on Machine Learning, vol. 37, pp. 957–966 (2015)
Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)
Li, Y., Xu, L., Tian, F., Jiang, L., Zhong, X., Chen, E.: Word embedding revisited: a new representation learning and explicit matrix factorization perspective. In: IJCAI International Joint Conference on Artificial Intelligence, pp. 3650–3656 (2015)
Tashu, T.M., Horváth, T.: Pair-wise: automatic essay evaluation using word mover’s distance. In: Proceedings of the 10th International Conference on Computer Supported Education, CSEDU, INSTICC, vol. 2, pp. 59–66. SciTePress (2018)
Shermis, M.D., Koch, C.M., Page, E.B., Keith, T.Z., Harrington, S.: Trait ratings for automated essay grading. Educ. Psychol. Measur. 62, 5–18 (2002)
Wang, X.B.: J. Educ. Behav. Stat. 30 (2005)
Zhang, L.: Review of handbook of automated essay evaluation: Current applications and new directions. Lang. Learn. Technol. 18, 65–69 (2014)
Ben-Simon, A., Bennett, R.E.: Toward more substantively meaningful automated essay scoring. J. Technol. Learn. Assess. 6(1) (2007)
Attali, Y., Burstein, J.: Automated essay scoring with e-rater® V.2. J. Technol. Learn. Assess. 4 (2006)
Cutrone, L., Chang, M.: Kinshuk: auto-assessor: computerized assessment system for marking student’s short-answers automatically. In: Proceedings of the IEEE International Conference on Technology for Education, pp. 81–88 (2011). https://doi.org/10.1109/T4E.2011.21
Foltz, P.W., Laham, D., Landauer, T.K.: Automated essay scoring: applications to educational technology. In: World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA) (1999)
Islam, M., Hoque, A.S.M.L.: Automated essay scoring using generalized latent semantic analysis. In: IEEE 13th International Conference on Computer and Information Technology, vol. 7, pp. 616–626 (2012)
Shermis, M.D., Burstein, J.: Automated essay scoring a cross-disciplinary perspective. Br. J. Math. Stat. Psychol. (2003)
Jin, C., He, B.: Utilizing latent semantic word representations for automated essay scoring. In: 12th International Conference on Ubiquitous Intelligence and Computing and IEEE 12th International Conference on Autonomic and Trusted Computing and IEEE 15th International Conference on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom) (2015)
Alikaniotis, D., Yannakoudakis, H., Rei, M.: Automatic text scoring using neural networks. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 715–725 (2016)
Taghipour, K., Ng, H.T.: A neural approach to automated essay scoring. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1882–1891. Association for Computational Linguistics (2016)
Jin, C., He, B., Xu, J.: A study of distributed semantic representations for automated essay scoring. In: Li, G., Ge, Y., Zhang, Z., Jin, Z., Blumenstein, M. (eds.) KSEM 2017. LNCS (LNAI), vol. 10412, pp. 16–28. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63558-3_2
Thanawala, P., Pareek, J., Shah, M.: OntoBAeval: ontology-based automatic evaluation of free-text response. In: 2014 IEEE Sixth International Conference on Technology for Education (2014)
Fauzi, M.A., Utomo, D.C., Setiawan, B.D., Pramukantoro, E.S.: Automatic essay scoring system using N-gram and cosine similarity for gamification based E-learning. In: Proceedings of the International Conference on Advances in Image Processing, ICAIP 2017, pp. 151–155. ACM, New York (2017)
Zupanc, K., Bosnifć, Z.: Automated essay evaluation with semantic analysis. Knowl.-Based Syst. 120, 118–132 (2017)
Yamamoto, M., Umemura, N., Kawano, H.: Automated essay scoring system based on rubric. In: Lee, R. (ed.) ACIT 2017. SCI, vol. 727, pp. 177–190. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-64051-8_11
Dumais, T.K., Landauer, S.: Latent semantic analysis. Scholarpedia 3(11), 4356 (2008)
Salton, G.: Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)
Porter, M.: The Porter Stemming Algorithm (1980)
Rajaraman, A., Ullman, J.D.: Mining of Massive Datasets. Cambridge University Press, New York (2011)
Islam, M., Latiful Hoque, A.S.M.: Automated essay scoring using generalized latent semantic analysis. In: International Conference on Computer and Information Technology (2010)
Atoum, I., Otoom, A.: Efficient hybrid semantic text similarity using wordnet and a corpus. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 7, 124–130 (2016)
Wan, S., Angryk, R.A.: Measuring semantic similarity using WordNet-based context vectors. In: IEEE International Conference on Systems, Man and Cybernetics (2007)
Zhuge, W., Hua, J.: WordNet-based way to identify Chinglish in automated essay scoring systems. In: International Symposium on Knowledge Acquisition and Modeling (2009)
Ewees, A.A., Eisa, M., Refaat, M.M.: Comparison of cosine similarity and k-NN automated essays scoring. Int. J. Adv. Res. Comput. Commun. Eng. 3 (2014)
Xia, P., Zhang, L., Li, F.: Learning similarity with cosine similarity ensemble. Inf. Sci. 307, 39–52 (2015)
Williamson, D.: A framework for implementing automated scoring. In: The Annual Meeting of the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME) (2009)
Clough, P., Stevenson, M.: Developing a corpus of plagiarised short answers. Lang. Resour. Eval. 45, 5–24 (2011)
Powers, D.M.W.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J. Mach. Learn. Technol. 2, 37–63 (2011)
Acknowledgements
The research has been supported by the European Union, co- financed by the European Social Fund (EFOP-3.6.2-16-2017-00013).
Supported by Telekom Innovation Laboratories (T-Labs), the Research and Development unit of Deutsche Telekom.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Tashu, T.M., Horváth, T. (2019). A Layered Approach to Automatic Essay Evaluation Using Word-Embedding. In: McLaren, B., Reilly, R., Zvacek, S., Uhomoibhi, J. (eds) Computer Supported Education. CSEDU 2018. Communications in Computer and Information Science, vol 1022. Springer, Cham. https://doi.org/10.1007/978-3-030-21151-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-21151-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-21150-9
Online ISBN: 978-3-030-21151-6
eBook Packages: Computer ScienceComputer Science (R0)