Abstract
Automated Essay Scoring (AES) is the NLP task of evaluating prose text, still scarcely explored in Portuguese. In this work, we present two AES strategies: the first with a deep neural network with two recurrent layers, and the second with a large number of handcrafted features. We apply our methods to evaluate essays from the ENEM exam with respect to five writing competencies. Overall, our feature-based system performs better in the first four, while the neural networks are better in the fifth one, which is also the hardest to grade accurately. In the aggregated score, our best model achieves a Quadratic Weighted Kappa of 0.752 and a Rooted Mean Squared Error of 100.0 when compared to human judgments, with scores ranging from 0 to 1000.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at https://www.kaggle.com/c/asap-aes.
- 2.
Exame Nacional de Ensino Médio, which serves as an entrance exam for most public universities in Brazil.
- 3.
- 4.
The baseline always has a QWK of zero because of the definition of the metric, which expects some variation in the results.
References
De Amorim, E.C.F., Veloso, A.: A multi-aspect analysis of automatic essay scoring for Brazilian Portuguese. In: Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, pp. 94–102 (2017)
Chen, H., He, B.: Automated essay scoring by maximizing human-machine agreement. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1741–1752 (2013)
Dikli, S.: An overview of automated scoring of essays. J. Technol. Learn. Assess. 5 (2006)
Dong, F., Zhang, Y.: Automatic features for essay scoring - an empirical study. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1072–1077 (2016)
Dong, F., Zhang, Y., Yang, J.: Attention-based recurrent convolutional neural network for automatic essay scoring. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 153–162 (2017)
Fonseca, E.R., Rosa, J.L.G., Aluísio, S.M.: Evaluating word embeddings and a revised corpus for part-of-speech tagging in Portuguese. J. Braz. Comput. Soc. 21(2) (2015)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. ArXiv e-prints, December 2014
Larkey, L.S.: Automatic essay grading using text categorization techniques. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1998 (1998), pp. 90–95. https://doi.org/10.1145/290941.290965
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
Perelman, L.: When “the state of the art” is counting words. Assess. Writ. 21, 104–111 (2014)
Taghipour, K., Ng, H.T.: A neural approach to automated essay scoring. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1882–1891 (2016)
Williamson, D.M.: A framework for implementing automated scoring. In: Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education (2009)
Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 180–189 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Fonseca, E., Medeiros, I., Kamikawachi, D., Bokan, A. (2018). Automatically Grading Brazilian Student Essays. In: Villavicencio, A., et al. Computational Processing of the Portuguese Language. PROPOR 2018. Lecture Notes in Computer Science(), vol 11122. Springer, Cham. https://doi.org/10.1007/978-3-319-99722-3_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-99722-3_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99721-6
Online ISBN: 978-3-319-99722-3
eBook Packages: Computer ScienceComputer Science (R0)