Skip to main content
SpringerLink
Log in

Ans2vec: A Scoring System for Short Answers

  • Conference paper
  • First Online:
The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2019) (AMLTA 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 921))

Abstract

Automatic scoring is a complex task in computational linguistics, particularly in an educational context. Sentences vectors (sent2vec) approaches affirmed their prosperity recently as favorable models for sentence representation. In this research, we propose an efficient and uncomplicated short answer grading model named Ans2vec. Skip-thought vector approach is used to convert both model and student’s answers into meaningful vectors to measure the similarity between them. Ans2vec model achieves promising results on three different benchmarking data sets. For Texas data set; Ans2vec achieves the best Pearson correlation value (0.63) compared to all related systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Gomaa, W.H., Fahmy, A.A.: Tapping into the power of automatic scoring. In: The Eleventh International Conference on Language Engineering, Egyptian Society of Language Engineering (ESOLEC) (2011)

    Google Scholar 

  2. Gomaa, W.H., Fahmy, A.A.: Arabic short answer scoring with effective feedback for students. Int. J. Comput. Appl. 86(2), 35–41 (2014)

    Google Scholar 

  3. Magooda, A.E., Zahran, M.A., Rashwan, M., Raafat, H.M., Fayek, M.B.: Vector based techniques for short answer grading. In: FLAIRS Conference, pp. 238–243, March 2016

    Google Scholar 

  4. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space (2013). arXiv preprint: arXiv:1301.3781

  5. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  6. Jiao, X., Wang, F., Feng, D.: Convolutional neural network for universal sentence embeddings. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2470–2481 (2018)

    Google Scholar 

  7. Socher, R., Huval, B., Manning, C.D., Ng, A.Y.: Semantic compositionality through recursive matrix-vector spaces. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1201–1211. Association for Computational Linguistics, July 2012

    Google Scholar 

  8. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)

    Google Scholar 

  9. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences (2014). arXiv preprint: arXiv:1404.2188

  10. dos Santos, C., Gatti, M.: Deep convolutional neural networks for sentiment analysis of short texts. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 69–78 (2014)

    Google Scholar 

  11. Kim, Y.: Convolutional neural networks for sentence classification (2014). arXiv preprint: arXiv:1408.5882

  12. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling (2014). arXiv preprint: arXiv:1412.3555

  13. Yin, W., Schütze, H.: Convolutional neural network for paraphrase identification. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 901–911 (2015)

    Google Scholar 

  14. Tan, M., Dos Santos, C., Xiang, B., Zhou, B.: Improved representation learning for question answer matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Long Papers, vol. 1, pp. 464–473 (2016)

    Google Scholar 

  15. Lin, Z., Feng, M., Santos, C.N.D., Yu, M., Xiang, B., Zhou, B., Bengio, Y.: A structured self-attentive sentence embedding (2017). arXiv preprint: arXiv:1703.03130

  16. Socher, R., Huang, E.H., Pennin, J., Manning, C.D., Ng, A.Y.: Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Advances in Neural Information Processing Systems, pp. 801–809 (2011)

    Google Scholar 

  17. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, pp. 1188–1196, January 2014

    Google Scholar 

  18. Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A., Fidler, S.: Skip-thought vectors. In: Advances in Neural Information Processing Systems, pp. 3294–3302 (2015)

    Google Scholar 

  19. Kenter, T., Borisov, A., de Rijke, M.: Siamese cbow: Optimizing word embeddings for sentence representations (2016). arXiv preprint: arXiv:1606.04640

  20. Hill, F., Cho, K., Korhonen, A.: Learning distributed representations of sentences from unlabelled data (2016). arXiv preprint: arXiv:1602.03483

  21. Pagliardini, M., Gupta, P., Jaggi, M.: Unsupervised learning of sentence embeddings using compositional n-gram features (2017). arXiv preprint: arXiv:1703.02507

  22. Wieting, J., Gimpel, K.: Revisiting recurrent networks for paraphrastic sentence embeddings (2017). arXiv preprint: arXiv:1705.00364

  23. Ziai, R., Ott, N., Meurers, D.: Short answer assessment: establishing links between research strands. In: Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pp. 190–200. Association for Computational Linguistics, June 2012

    Google Scholar 

  24. Burrows, S., Gurevych, I., Stein, B.: The eras and trends of automatic short answer grading. Int. J. Artif. Intell. Educ. 25(1), 60–117 (2015)

    Article  Google Scholar 

  25. Leacock, C., Chodorow, M.: C-rater: automated scoring of short-answer questions. Comput. Humanit. 37(4), 389–405 (2003)

    Article  Google Scholar 

  26. Rosé, C.P., Roque, A., Bhembe, D., VanLehn, K.: A hybrid approach to content analysis for automatic essay grading. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: Companion Volume of the Proceedings of HLT-NAACL 2003–Short papers, vol. 2, pp. 88–90. Association for Computational Linguistics, May 2003

    Google Scholar 

  27. Mitchell, T., Russell, T., Broomhead, P., Aldridge, N.: Towards robust computerised marking of free-text responses. In: Proceedings of the Sixth International Computer Assisted Assessment Conference. Loughborough University, Loughborough, UK (2002)

    Google Scholar 

  28. Pulman, S.G., Sukkarieh, J.Z.: Automatic short answer marking. In: Proceedings of the Second Workshop on Building Educational Applications Using NLP, pp. 9–16. Association for Computational Linguistics, June 2005

    Google Scholar 

  29. Mohler, M., Bunescu, R., Mihalcea, R.: Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 752–762. Association for Computational Linguistics, June 2011

    Google Scholar 

  30. Hewlett Foundation: Automated student assessment prize: phase two – short answer scoring. Kaggle Competition (2012)

    Google Scholar 

  31. Dzikovska, M.O., Nielsen, R.D., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., Clark, P., Dagan, I., Dang, H.T.: SemEval-2013 task 7: the joint student response analysis and eighth recognizing textual entailment challenge. In: Diab, M., Baldwin, T., Baroni, M. (eds.) Proceedings of the 2nd Joint Conference on Lexical and Computational Semantics, Atlanta, pp. 1–12 (2013)

    Google Scholar 

  32. Burstein, J., Kaplan, R., Wolff, S., Lu, C.: Using lexical semantic techniques to classify free responses. In: Viegas, E. (ed.) Proceedings of the ACL SIGLEX Workshop on Breadth and Depth of Semantic Lexicons, Santa Cruz, pp. 20–29. Association for Computational Linguistics (1996)

    Google Scholar 

  33. Callear, D., Jerrams-Smith, J., Soh, V.: CAA of short non-MCQ answers. In: Danson, M., Eabry, C. (eds.) Proceedings of the 5th Computer Assisted Assessment Conference, pp. 1–14. Loughborough University, Loughborough (2001)

    Google Scholar 

  34. Wang, H.-C., Chang, C.-Y., Li, T.-Y.: Assessing creative problem-solving with automated text grading. Comput. Educ. 51(4), 1450–1466 (2008)

    Article  Google Scholar 

  35. Bachman, L.F., Carr, N., Kamei, G., Kim, M., Pan, M.J., Salvador, C., Sawaki, Y.: A reliable approach to automatic assessment of short answer free responses. In: Tseng, S.C., Chen, T.E., Liu, Y.F. (eds.) Proceedings of the 19th International Conference on Computational Linguistics, COLING 2002, Taipei, vol. 2, pp. 1–4. Association for Computational Linguistics (2002)

    Google Scholar 

  36. Sima, D., Schmuck, B., Szöll, S., Miklós, A.: Intelligent short text assessment in eMax. In: Rudas, I.J., Fodor, J., Kacprzyk, J. (eds.) Towards Intelligent Engineering and Information Technology. Studies in Computational Intelligence, vol. 243, pp. 435–445. Springer, Heidelberg (2009)

    Google Scholar 

  37. Cutrone, L., Chang, M., Kinshuk: Auto-assessor: computerized assessment system for marking student’s short-answers automatically. In: Narayanaswamy, N.S., Krishnan, M.S., Kinshuk, Srinivasan, R. (eds.) Proceedings of the 3rd IEEE International Conference on Technology for Education, Chennai, pp. 81–88. IEEE (2011)

    Google Scholar 

  38. Siddiqi, R., Harrison, C.J.: A systematic approach to the automated marking of short-answer questions. In: Anis, M.K., Khan, M.K., Zaidi, S.J.H. (eds.) Proceedings of the 12th International Multitopic Conference, Karachi, pp. 329–332. IEEE (2008)

    Google Scholar 

  39. Alfonseca, E., Pérez, D.: Automatic assessment of open ended questions with a BLEU-inspired algorithm and shallow NLP. In: Vicedo, J., Martínez-Barco, P., Muńoz, R., Saiz Noeda, M. (eds.) Advances in Natural Language Processing. Lecture Notes in Computer Science, vol. 3230, pp. 25–35. Springer, Berlin (2004)

    Google Scholar 

  40. Pérez-Marín, D., Pascual-Nieto, I.: Willow: a system to automatically assess students’ freetext answers by using a combination of shallow NLP techniques. Int. J. Contin. Eng. Educ. Life Long Learn. 21(2), 155–169 (2011)

    Article  Google Scholar 

  41. Bukai, O., Pokorny, R., Haynes, J.: An automated short-free-text scoring system: development and assessment. In: Proceedings of the 20th Interservice/Industry Training, Simulation, and Education Conference, pp. 1–11. National Training and Simulation Association (2006)

    Google Scholar 

  42. Gütl, C.: e-Examiner: towards a fully-automatic knowledge assessment tool applicable in adaptive e-learning systems. In: Ghassib, P.H. (ed.) Proceedings of the 2nd International Conference on Interactive Mobile and Computer Aided Learning, pp. 1–10. Amman (2007)

    Google Scholar 

  43. Bailey, S., Meurers, D.: Diagnosing meaning errors in short answers to reading comprehension questions. In: Tetreault, J., Burstein, J., De Felice, R. (eds.) Proceedings of the 3rd ACL Workshop on Innovative Use of NLP for Building Educational Applications, Columbus, pp. 107–115. Association for Computational Linguistics (2008)

    Google Scholar 

  44. Meurers, D., Ziai, R., Ott, N., Bailey, S.M.: Integrating parallel analysis modules to evaluate the meaning of answers to reading comprehension questions. Int. J. Contin. Eng. Educ. Life-Long Learn. 21(4), 355–369 (2011)

    Article  Google Scholar 

  45. Gomaa, W.H., Fahmy, A.A.: Short answer grading using string similarity and corpus-based similarity. Int. J. Adv. Comput. Sci. and Appl. (IJACSA) 3(11), 115–121 (2012)

    Google Scholar 

  46. Gomaa, W.H., Fahmy, A.A.: Automatic scoring for answers to Arabic test questions. Comput. Speech Lang. 28(4), 833–857 (2014)

    Article  Google Scholar 

  47. Sultan, M.A., Salazar, C., Sumner, T.: Fast and easy short answer grading with high accuracy. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1070–1075 (2016)

    Google Scholar 

  48. Gomaa, W.H., Fahmy, A.A.: A survey of text similarity approaches. Int. J. Comput. Appl. 68(13), 13–18 (2013)

    Google Scholar 

  49. Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks (2015). arXiv preprint: arXiv:1503.00075

  50. Dzikovska, M.O., Nielsen, R.D., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., Dang, H.T.: Semeval-2013 task 7: the joint student response analysis and 8th recognizing textual entailment challenge. North Texas State Univ Denton (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beni-Suef University, Beni-Suef, Egypt

    Wael Hassan Gomaa

  2. Cairo University, Cairo, Egypt

    Aly Aly Fahmy

Authors
  1. Wael Hassan Gomaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aly Aly Fahmy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wael Hassan Gomaa .

Editor information

Editors and Affiliations

  1. Faculty of Computers and Information, Cairo University, Giza, Egypt

    Aboul Ella Hassanien

  2. Faculty of Computers and Information, Benha University, Benha, Egypt

    Ahmad Taher Azar

  3. School of Computing, Science and Engineering, University of Salford, Salford, Greater Manchester, UK

    Tarek Gaber

  4. Department of Computer Science and Engineering, School of Computing and IT, Faculty of Engineering, Manipal University Jaipur, Jaipur, Rajasthan, India

    Roheet Bhatnagar

  5. Faculty of Computer and Information Science, Ain Shams University, Cairo, Egypt

    Mohamed F. Tolba

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gomaa, W.H., Fahmy, A.A. (2020). Ans2vec: A Scoring System for Short Answers. In: Hassanien, A., Azar, A., Gaber, T., Bhatnagar, R., F. Tolba, M. (eds) The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2019). AMLTA 2019. Advances in Intelligent Systems and Computing, vol 921. Springer, Cham. https://doi.org/10.1007/978-3-030-14118-9_59

Download citation

Publish with us

Policies and ethics

Search

Navigation