Abstract
The Automatic Short Answer Scoring (ASAS) system is one of the tools that can be used to conduct assessment process on e-learning system. One of the methods applied in the ASAS system is a method for measuring similarities between the reference and student answers. There are two issues to be considered in the assessment process using this method. First, this method should be able to provide a variety of reference answers that can handle the diversity of student answers. Secondly, this method should be able to provide an accurate sentence similarity between the reference answers and student answers. Therefore, two methods are proposed to solve both problems. The first method is to generate a variety of reference answers automatically using Maximum Marginal Relevance (MMR) method, which obtains an accuracy of 91.95%. The second method is to measure accurately sentence similarity between student answers and reference answers that have significantly different length using GAN-LCS. The performance of the proposed method shows an improvement of the Root Mean Square Error (RMSE) value of 0.884 and a correlation value of 0.468.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adhitia, R., Purwarianti, A., & Bandung, I. T. (2009). Automated essay grading system using SVM and LSA for essay answers in Indonesian. Journal of Information Systems, 5(1), 33–41.
Bachman, L. F., et al. (2002). A reliable approach to automatic assessment of short answer free responses. Proceedings of the 19th International Conference on Computational Linguistics, 2, 1–4.
Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60–117.
Carbonell, J., & Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ‘98 (pp. 335–336).
Evans, J. D. (1996). Straightforward statistics for the behavioral sciences. Pacific Grove: Brooks/Cole Pub. Co.
Gomaa, W. H., & Fahmy, A. A. (2012). Short answer grading using string similarity and corpus-based similarity. International Journal of Advanced Computer Science and Applications, 3(11), 115–121.
Jayashankar, S., & Sridaran, R. (2017). Superlative model using word cloud for short answers evaluation in eLearning. Education and Information Technologies, 22(5), 2383–2402.
Klein, R., Kyrilov, A., & Tokman, M. (2011). Automated assessment of short free-text responses in computer science using latent semantic analysis. In Proceedings of the 16th annual joint conference on Innovation and technology in computer science education (pp. 158–162).
Leacock, C., & Chodorow, M. (2003). C-rater: automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.
Mohler, M., & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL ‘09 (pp. 567–575).
Mohler, M., Bunescu, R., & Mihalcea, R. (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 752–762).
Noorbehbahani, F., & Kardan, A. A. (2011). The automatic assessment of free text answers using a modified BLEU algorithm. Computers & Education, 56(2), 337–345.
Pérez, D., & Alfonseca, E. (2005). Application of the Bleu algorithm for recognising textual entailments. In Workshop Recognising Textual Entailment (pp. 1–4).
Rodrigues, F., & Araújo, L. (2012). Automatic assessment of short free text answers. In CSEDU 2012 - Proceedings of the 4th International Conference on Computer Supported Education (pp. 50–57).
Senthil Kumaran, V., & Sankar, A. (2015). Towards an automated system for short-answer assessment using ontology mapping. International Arab Journal of e-Technology, 4, 17–25.
Shermis, M. D. (2015). Contrasting state-of-the-art in the machine scoring of short-form constructed responses. Educational Assessment, 20(1), 46–65.
Siddiqi, R., Harrison, C. J., & Siddiqi, R. (2010). Improving teaching and learning through automated short-answer marking. IEEE Transactions on Learning Technologies, 3(3), 237–249.
Spiliotopoulou-Papantoniou, V. (2007). Models of the universe: children’s experiences and evidence from the history of science. Science Education, 16, 801–833.
Sukkarieh, J., & Blackmore, J. (2009). c-rater: Automatic content scoring for short constructed responses. In Proceedings of the Twenty-Second International FLAIRS Conference (pp. 290–295).
Sultan, M. A., Salazar, C., & Sumner, T. (2016). Fast and easy short answer grading with high accuracy. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1070–1075).
Wolska, M., Horbach, A., & Palmer, A. (2014). Computer-assisted scoring of short responses: the efficiency of a clustering-based approach in a real-life task. Advances in Natural Language Processing, 298–310.
Xi, Y., & Liang, W. (2011). Automated computer-based CET4 essay scoring system. In Proceedings - PACCS 2011: 2011 3rd Pacific-Asia Conference on Circuits, Communications and System.
Ziai, R., Ott, N., & Meurers, D. (2012). Short answer assessment: Establishing links between research strands. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 190–200).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pribadi, F.S., Permanasari, A.E. & Adji, T.B. Short answer scoring system using automatic reference answer generation and geometric average normalized-longest common subsequence (GAN-LCS). Educ Inf Technol 23, 2855–2866 (2018). https://doi.org/10.1007/s10639-018-9745-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-018-9745-z