Skip to main content
SpringerLink
Log in

Superlative model using word cloud for short answers evaluation in eLearning

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Teachers are thrown open to abundance of free text answers which are very daunting to read and evaluate. Automatic assessments of open ended answers have been attempted in the past but none guarantees 100 % accuracy. In order to deal with the overload involved in this manual evaluation, a new tool becomes necessary. The unique superlative model discussed in this paper aims at providing improved accuracy by constructing word clouds. The model uses appropriate semantics with a visual appeal to partially automate free text evaluation. The model was applied at a K-12 school setup where the average human agreement rate was found to be 98 % and the accuracy score deviation from the mean was 2.82. This tool can be cast-off at any level starting from K-12 to higher education to evolve the way we view and evaluate answers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Chart 1

Similar content being viewed by others

Notes

  1. http://www.kaggle.com

  2. Official Website of National Academy For Learning, India - http://www.nafl.in/

  3. Official Website of Cambridge International Examinations UK, http://www.cie.org.uk/programmes-and-qualifications/cambridge-secondary-2/cambridge-igcse/

References

  • Ahearn, L. M. (2013). Commentary: keywords as a literacy practice in the history of anthropological theory. American Ethnologist, 40(1), 6–12.

    Article  Google Scholar 

  • Bailey, S. & Meurers, D., (2008). Diagnosing meaning errors in short answers to reading comprehension questions. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 107–115). Association for Computational Linguistics.

  • Baralt, M., Pennestri, S., & Selvandin, M. (2011). Action research: using wordles to teach foreign language writing. Language Learning & Technology, 15(2), 12–22.

    Google Scholar 

  • Behar-Horenstein, L. S., & Niu, L. (2011). Teaching critical thinking skills in higher education: a review of the literature. Journal of College Teaching and Learning, 8(2), 25.

    Google Scholar 

  • Bielenberg, K., (2005). Groups in social software: Utilizing tagging to integrate individual contexts for social navigation (Doctoral dissertation, Universität Bremen).

  • Blin, F., & Munro, M. (2008). Why hasn’t technology disrupted academics’ teaching practices? understanding resistance to change through the lens of activity theory. Computers & Education, 50(2), 475–490.

    Article  Google Scholar 

  • Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals: Cognitive Domain. Longman.

  • Brin, S., Motwani, R., Ullman, J.D. & Tsur, S., (1997). Dynamic itemset counting and implication rules for market basket data. In ACM SIGMOD Record (Vol. 26, No. 2, pp. 255–264). ACM.

  • Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60–117.

    Article  Google Scholar 

  • Burstein, J., Wolff, S. & Lu, C., (1999). Using lexical semantic techniques to classify free-responses. In Breadth and depth of semantic lexicons (pp. 227–244). Springer Netherlands.

  • Butcher, P. G., & Jordan, S. E. (2010). A comparison of human and computer marking of short free-text student responses. Computers & Education, 55(2), 489–499.

    Article  Google Scholar 

  • Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug), 2493–2537.

    MATH  Google Scholar 

  • Concetta, A. DePaolo, Wilkinson, K. (2014). get your head into the clouds: using word clouds for analyzing qualitative assessment data. TechTrends-May/June 2014, Springer, Volume 58, Number 3.

  • Cook, K. A. & Thomas, J. J. (2005). Illuminating the path: The research and development agenda for visual analytics (No. PNNL-SA-45230). Pacific Northwest National Laboratory (PNNL), Richland, WA (US).

  • Cook, K., Earnshaw, R., & Stasko, J. (2007). Guest editors’ introduction: Discovering the unexpected. IEEE Computer Graphics and Applications, 27(5), 15–19.

    Article  Google Scholar 

  • deNoyelles, A., & Reyes-Foster, B. (2015). Using word clouds in online discussions to support critical thinking and engagement. Online Learning, 19(4), n4.

    Article  Google Scholar 

  • DePaolo, C. A., & Wilkinson, K. (2014). Get your head into the clouds: using word clouds for analyzing qualitative assessment data. TechTrends, 58(3), 38–44.

    Article  Google Scholar 

  • Dzikovska, M. O., Nielsen, R. D., & Leacock, C. (2016). The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications. Language Resources and Evaluation, 50(1), 67–93.

    Article  Google Scholar 

  • Feinberg, J. (2010). Wordle. In J. Steele & N. Iliinsky (Eds.), Beautiful visualization (pp. 37–58). Sebastopol: O’Reilly.

    Google Scholar 

  • Feinerer, I., Hornik, K., Wallace, M. & Hornik, M.K., (2016). Package ‘wordnet’. https://CRAN.R-project.org/package=wordnet, Accessed 21 January 2016.

  • Fellows (2013). https://cran.r-project.org/web/packages/wordcloud/index.html, Accessed 22 January 2016.

  • Gütl, C. (2007). e-Examiner: towards a fully-automatic knowledge assessment tool applicable in adaptive e-learning systems. In Proceedings of the 2nd international conference on interactive mobile and computer aided learning (pp. 1–10).

  • Hahn, M. & Meurers, D. (2012). Evaluating the meaning of answers to reading comprehension questions a semantics-based approach. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 326–336). Association for Computational Linguistics.

  • Hahsler, M., Grün, B. & Hornik, K. (2005). A computational environment for mining association rules and frequent item sets.

  • Huisman, R., Miller, W., & Trinoskey, J. (2011). We’ve Wordled, have you? digital images in the library classroom. College & Research Libraries News, 72(9), 522–526.

    Article  Google Scholar 

  • Ingo, F & Kurt, H. (2015). tm: Text Mining Package. R package version 0.6-2. http://CRAN.R-project.org/package=tm, Accessed 21 January 2016.

  • Ingoley, S., & Bakal, J. W. (2012). Use of fuzzy logic in evaluating students’ learning achievement. International Journal on Advanced Computer Engineering and Communication Technology (IJACECT), 1(2), 47–54.

    Google Scholar 

  • Jablon, J.R. & Wilkinson, M. (2006). Using engagement strategies to facilitate children’s learning and success. YC Young Children 12-16.

  • Jimoyiannis, A., & Komis, V. (2007). Examining teachers’ beliefs about ICT in education: implications of a teacher preparation programme. Teacher Development, 11(2), 149–173.

    Article  Google Scholar 

  • Kao, A. & Poteet, S.R. eds. (2007). Natural language processing and text mining. Springer Science & Business Media.

  • Kaptein, R., Hiemstra, D. & Kamps, J., (2010). How different are language models andword clouds?. In European Conference on Information Retrieval (pp. 556–568). Springer Berlin Heidelberg.

  • Kolb, A, Y. (2005). The Kolb Learning Style Inventory—Version 3.1 2005 Technical Specifications. Boston, MA: Hay Resource Direct, 200.

  • Kragten, J. (1994). Tutorial review. calculating standard deviations and confidence intervals with a universally applicable spreadsheet technique. Analyst, 119(10), 2161–2165.

    Article  Google Scholar 

  • Leacock, C., & Chodorow, M. (2003). C-rater: automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.

    Article  Google Scholar 

  • Leek, J, Peng, R. & Irizarry, R. (2013). Simply statistics”, http://www.r-bloggers.com/a-wordcloud-comparison-of-the-2011-and-2012-sotu/#, Accessed 21 January 2016.

  • Liu, V. & Curran, J. R. (2006). Web Text Corpus for Natural Language Processing. In EACL.

  • Maglogiannis, I. G. (2007). Emerging artificial intelligence applications in computer engineering: real word AI systems with applications in eHealth, HCI, information retrieval and pervasive technologies (Vol. 160). Ios Press.

  • McNaught, C., & Lam, P. (2010). Using Wordle as a supplementary research tool. The Qualitative Report, 15(3), 630.

    Google Scholar 

  • Meyer, D., Hornik, K. & Feinerer, I. (2008). Text mining infrastructure in R. Journal of Statistical Software 25(5), pp.1-54. ISSN 1548–7660. URL http://www.jstatsoft.org/v25/i05, Accessed 21 April 2016.

  • Miley, F., & Read, A. (2012). Using word clouds to develop proactive learners. Journal of the Scholarship of Teaching and Learning, 11(2), 91–110.

    Google Scholar 

  • Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11), 39–41.

    Article  Google Scholar 

  • Mitchell, T., Russell, T., Broomhead, P. & Aldridge, N. (2002). Towards robust computerised marking of free-text responses.

  • Mohler, M. & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (pp. 567–575). Association for Computational Linguistics.

  • Nickell, J. (2012). Word clouds in math classrooms. Mathematics Teaching in the Middle School, 17(9), 564–566.

    Article  Google Scholar 

  • Pérez, D., Gliozzo, A. M., Strapparava, C., Alfonseca, E., Rodríguez, P. & Magnini, B. (2005). Automatic assessment of students’ free-text answers underpinned by the combination of a BLEU-inspired algorithm and latent semantic analysis. In FLAIRS conference (pp. 358–363).

  • Perry, L. (2012). Using word clouds to teach about speaking style. Communication Teacher, 26(4), 220–223.

    Article  Google Scholar 

  • Pulman, S.G. & Sukkarieh, J. Z. (2005). Automatic short answer marking. In Proceedings of the second workshop on Building Educational Applications Using NLP (pp. 9–16). Association for Computational Linguistics.

  • Ramsden, A. & Bate, A. (2008). Using word clouds in teaching and learning.

  • Sag I. A., Baldwin, T., Bond, F., Copestake, A. & Flickinger, D. (2002). Multiword expressions: a pain in the neck for NLP. In International Conference on Intelligent Text Processing and Computational Linguistics (pp. 1–15). Springer Berlin Heidelberg.

  • Siddiqi, R. (2010). Improving learning and teaching through automated short answer marking, complete thesis, https://www.escholar.manchester.ac.uk/api/datastream?publicationPid=uk-ac-manscw:90867&datastreamId=FULL-TEXT.PDF, Accessed 28 July 2016.

  • Siddiqi, R., Harrison, C. J., & Siddiqi, R. (2010). Improving teaching and learning through automated short-answer marking. IEEE Transactions on Learning Technologies, 3(3), 237–249.

    Article  Google Scholar 

  • Sukkarieh, J. Z., Pulman, S.G. & Raikes, N. (2003). Auto-marking: using computational linguistics to score short, free text responses. In the annual conference of the International Association for Educational Assessment (IAEA), Manchester, UK.

  • Tsui, L. (2002). Fostering critical thinking through effective pedagogy: evidence from four institutional case studies. The Journal of Higher Education, 73(6), 740–763.

    Google Scholar 

  • Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.

    Article  Google Scholar 

  • Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 366(1881), 3717–3725.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science, National Academy for Learning, Bangalore, Karnataka, India

    Shailaja Jayashankar

  2. Faculty of Computer Applications, Marwadi University, Rajkot, Gujarat, India

    R. Sridaran

Authors
  1. Shailaja Jayashankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Sridaran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shailaja Jayashankar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jayashankar, S., Sridaran, R. Superlative model using word cloud for short answers evaluation in eLearning. Educ Inf Technol 22, 2383–2402 (2017). https://doi.org/10.1007/s10639-016-9547-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-016-9547-0

Keywords

Search

Navigation