Abstract
Teachers are thrown open to abundance of free text answers which are very daunting to read and evaluate. Automatic assessments of open ended answers have been attempted in the past but none guarantees 100 % accuracy. In order to deal with the overload involved in this manual evaluation, a new tool becomes necessary. The unique superlative model discussed in this paper aims at providing improved accuracy by constructing word clouds. The model uses appropriate semantics with a visual appeal to partially automate free text evaluation. The model was applied at a K-12 school setup where the average human agreement rate was found to be 98 % and the accuracy score deviation from the mean was 2.82. This tool can be cast-off at any level starting from K-12 to higher education to evolve the way we view and evaluate answers.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Official Website of National Academy For Learning, India - http://www.nafl.in/
Official Website of Cambridge International Examinations UK, http://www.cie.org.uk/programmes-and-qualifications/cambridge-secondary-2/cambridge-igcse/
References
Ahearn, L. M. (2013). Commentary: keywords as a literacy practice in the history of anthropological theory. American Ethnologist, 40(1), 6–12.
Bailey, S. & Meurers, D., (2008). Diagnosing meaning errors in short answers to reading comprehension questions. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 107–115). Association for Computational Linguistics.
Baralt, M., Pennestri, S., & Selvandin, M. (2011). Action research: using wordles to teach foreign language writing. Language Learning & Technology, 15(2), 12–22.
Behar-Horenstein, L. S., & Niu, L. (2011). Teaching critical thinking skills in higher education: a review of the literature. Journal of College Teaching and Learning, 8(2), 25.
Bielenberg, K., (2005). Groups in social software: Utilizing tagging to integrate individual contexts for social navigation (Doctoral dissertation, Universität Bremen).
Blin, F., & Munro, M. (2008). Why hasn’t technology disrupted academics’ teaching practices? understanding resistance to change through the lens of activity theory. Computers & Education, 50(2), 475–490.
Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals: Cognitive Domain. Longman.
Brin, S., Motwani, R., Ullman, J.D. & Tsur, S., (1997). Dynamic itemset counting and implication rules for market basket data. In ACM SIGMOD Record (Vol. 26, No. 2, pp. 255–264). ACM.
Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60–117.
Burstein, J., Wolff, S. & Lu, C., (1999). Using lexical semantic techniques to classify free-responses. In Breadth and depth of semantic lexicons (pp. 227–244). Springer Netherlands.
Butcher, P. G., & Jordan, S. E. (2010). A comparison of human and computer marking of short free-text student responses. Computers & Education, 55(2), 489–499.
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug), 2493–2537.
Concetta, A. DePaolo, Wilkinson, K. (2014). get your head into the clouds: using word clouds for analyzing qualitative assessment data. TechTrends-May/June 2014, Springer, Volume 58, Number 3.
Cook, K. A. & Thomas, J. J. (2005). Illuminating the path: The research and development agenda for visual analytics (No. PNNL-SA-45230). Pacific Northwest National Laboratory (PNNL), Richland, WA (US).
Cook, K., Earnshaw, R., & Stasko, J. (2007). Guest editors’ introduction: Discovering the unexpected. IEEE Computer Graphics and Applications, 27(5), 15–19.
deNoyelles, A., & Reyes-Foster, B. (2015). Using word clouds in online discussions to support critical thinking and engagement. Online Learning, 19(4), n4.
DePaolo, C. A., & Wilkinson, K. (2014). Get your head into the clouds: using word clouds for analyzing qualitative assessment data. TechTrends, 58(3), 38–44.
Dzikovska, M. O., Nielsen, R. D., & Leacock, C. (2016). The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications. Language Resources and Evaluation, 50(1), 67–93.
Feinberg, J. (2010). Wordle. In J. Steele & N. Iliinsky (Eds.), Beautiful visualization (pp. 37–58). Sebastopol: O’Reilly.
Feinerer, I., Hornik, K., Wallace, M. & Hornik, M.K., (2016). Package ‘wordnet’. https://CRAN.R-project.org/package=wordnet, Accessed 21 January 2016.
Fellows (2013). https://cran.r-project.org/web/packages/wordcloud/index.html, Accessed 22 January 2016.
Gütl, C. (2007). e-Examiner: towards a fully-automatic knowledge assessment tool applicable in adaptive e-learning systems. In Proceedings of the 2nd international conference on interactive mobile and computer aided learning (pp. 1–10).
Hahn, M. & Meurers, D. (2012). Evaluating the meaning of answers to reading comprehension questions a semantics-based approach. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 326–336). Association for Computational Linguistics.
Hahsler, M., Grün, B. & Hornik, K. (2005). A computational environment for mining association rules and frequent item sets.
Huisman, R., Miller, W., & Trinoskey, J. (2011). We’ve Wordled, have you? digital images in the library classroom. College & Research Libraries News, 72(9), 522–526.
Ingo, F & Kurt, H. (2015). tm: Text Mining Package. R package version 0.6-2. http://CRAN.R-project.org/package=tm, Accessed 21 January 2016.
Ingoley, S., & Bakal, J. W. (2012). Use of fuzzy logic in evaluating students’ learning achievement. International Journal on Advanced Computer Engineering and Communication Technology (IJACECT), 1(2), 47–54.
Jablon, J.R. & Wilkinson, M. (2006). Using engagement strategies to facilitate children’s learning and success. YC Young Children 12-16.
Jimoyiannis, A., & Komis, V. (2007). Examining teachers’ beliefs about ICT in education: implications of a teacher preparation programme. Teacher Development, 11(2), 149–173.
Kao, A. & Poteet, S.R. eds. (2007). Natural language processing and text mining. Springer Science & Business Media.
Kaptein, R., Hiemstra, D. & Kamps, J., (2010). How different are language models andword clouds?. In European Conference on Information Retrieval (pp. 556–568). Springer Berlin Heidelberg.
Kolb, A, Y. (2005). The Kolb Learning Style Inventory—Version 3.1 2005 Technical Specifications. Boston, MA: Hay Resource Direct, 200.
Kragten, J. (1994). Tutorial review. calculating standard deviations and confidence intervals with a universally applicable spreadsheet technique. Analyst, 119(10), 2161–2165.
Leacock, C., & Chodorow, M. (2003). C-rater: automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.
Leek, J, Peng, R. & Irizarry, R. (2013). Simply statistics”, http://www.r-bloggers.com/a-wordcloud-comparison-of-the-2011-and-2012-sotu/#, Accessed 21 January 2016.
Liu, V. & Curran, J. R. (2006). Web Text Corpus for Natural Language Processing. In EACL.
Maglogiannis, I. G. (2007). Emerging artificial intelligence applications in computer engineering: real word AI systems with applications in eHealth, HCI, information retrieval and pervasive technologies (Vol. 160). Ios Press.
McNaught, C., & Lam, P. (2010). Using Wordle as a supplementary research tool. The Qualitative Report, 15(3), 630.
Meyer, D., Hornik, K. & Feinerer, I. (2008). Text mining infrastructure in R. Journal of Statistical Software 25(5), pp.1-54. ISSN 1548–7660. URL http://www.jstatsoft.org/v25/i05, Accessed 21 April 2016.
Miley, F., & Read, A. (2012). Using word clouds to develop proactive learners. Journal of the Scholarship of Teaching and Learning, 11(2), 91–110.
Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11), 39–41.
Mitchell, T., Russell, T., Broomhead, P. & Aldridge, N. (2002). Towards robust computerised marking of free-text responses.
Mohler, M. & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (pp. 567–575). Association for Computational Linguistics.
Nickell, J. (2012). Word clouds in math classrooms. Mathematics Teaching in the Middle School, 17(9), 564–566.
Pérez, D., Gliozzo, A. M., Strapparava, C., Alfonseca, E., Rodríguez, P. & Magnini, B. (2005). Automatic assessment of students’ free-text answers underpinned by the combination of a BLEU-inspired algorithm and latent semantic analysis. In FLAIRS conference (pp. 358–363).
Perry, L. (2012). Using word clouds to teach about speaking style. Communication Teacher, 26(4), 220–223.
Pulman, S.G. & Sukkarieh, J. Z. (2005). Automatic short answer marking. In Proceedings of the second workshop on Building Educational Applications Using NLP (pp. 9–16). Association for Computational Linguistics.
Ramsden, A. & Bate, A. (2008). Using word clouds in teaching and learning.
Sag I. A., Baldwin, T., Bond, F., Copestake, A. & Flickinger, D. (2002). Multiword expressions: a pain in the neck for NLP. In International Conference on Intelligent Text Processing and Computational Linguistics (pp. 1–15). Springer Berlin Heidelberg.
Siddiqi, R. (2010). Improving learning and teaching through automated short answer marking, complete thesis, https://www.escholar.manchester.ac.uk/api/datastream?publicationPid=uk-ac-manscw:90867&datastreamId=FULL-TEXT.PDF, Accessed 28 July 2016.
Siddiqi, R., Harrison, C. J., & Siddiqi, R. (2010). Improving teaching and learning through automated short-answer marking. IEEE Transactions on Learning Technologies, 3(3), 237–249.
Sukkarieh, J. Z., Pulman, S.G. & Raikes, N. (2003). Auto-marking: using computational linguistics to score short, free text responses. In the annual conference of the International Association for Educational Assessment (IAEA), Manchester, UK.
Tsui, L. (2002). Fostering critical thinking through effective pedagogy: evidence from four institutional case studies. The Journal of Higher Education, 73(6), 740–763.
Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 366(1881), 3717–3725.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jayashankar, S., Sridaran, R. Superlative model using word cloud for short answers evaluation in eLearning. Educ Inf Technol 22, 2383–2402 (2017). https://doi.org/10.1007/s10639-016-9547-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-016-9547-0