Abstract
C-rater is an automated scoringengine that has been developed to scoreresponses to content-based short answerquestions. It is not simply a stringmatching program – instead it uses predicateargument structure, pronominal reference,morphological analysis and synonyms to assignfull or partial credit to a short answerquestion. C-rater has been used in two studies:National Assessment for Educational Progress(NAEP) and a statewide assessment in Indiana.In both studies, c-rater agreed with humangraders about 84% of the time.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abney S. (1996). Partial Parsing via Finite-State Cascades. Proceedings of the ESSLLI '96 Robust Parsing Workshop.
Burstein J., Wolff S., Lu C. (1999). Using Lexical Semantic Techniques to Classify Free-responses. In Ide N. and Veronis J. (eds.), The Depth and Breadth of Semantic Lexicons, Kluwer Academic Press.
Burstein J. (2003). The E-rater® Scoring Engine: Automated Essay Scoring with Natural Language Processing. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.
Cormen T.H., Leiserson C.E., Rivest R.L., Stein C. (2001). Introduction to Algorithms, Second Edition. The MIT Press, Cambridge, MA.
Elliott S. (2003) Intellemetric: From Here to Validity. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.
Fellbaum C. (1998). WordNet: An Electronic Lexical Database. MIT Press, Cambridge.
Fleiss J.L. (1981). Statistical Methods for Rates and Proportions. John Wiley & Sons, New York. pp. 212–236.
Landauer T.K., Laham D., Foltz P. (2003). Automated Scoring and Annotation of Essays with Intelligent Essay Assessor™. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.
Landauer T.K., Laham D., Rehder B., Schreiner M.E. (1997). How Well Can Passage Meaning be Derived Without Using Word Order? A Comparison of Latent Semantic Analysis and Humans. In Shafto M.G. and Langley P. (eds.), Proceedings of the 19th Anual Meeting of the Cognitive Science Society, Lawrence Erlbaum, Mahwah, NJ, pp. 412–417.
Kaplan R.M., Bennett R.E. (1994). Using the Free-response Scoring Tool to Automatically Score the Formulating-hypotheses Item. ETS Research Report 04–08.
Kud J.M., Krupka G.R., Rau L.F. (1994). Methods for Categorizing Short Answer Responses. Proceedings of the Educational Testing Service Conference on Natural Language Processing Techniques and Technology in Education and Assessment. Princeton, NJ.
Lin D. (1998). Automatic Retrieval and Clustering of SimilarWords. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistic, Montreal, pp. 898–904.
Morton T.S. (2000). Coreference for NLP applications. Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, Hong Kong.
Salton G., Wong A., Yang C.S. (1975). A Vector Space Model for Automatic Indexing. Communications of the ACM, 18/11, pp. 613–620.
Sandene B., Bennett R., Braswell J., Oranje A. (Forthcoming). Mathematics Online Study: Final Report. National Center for Education Statistics, Washington, DC.
Vigilante R. (1999). Online Computer Scoring of Constructed-response Questions. Journal of Information Technology Impact, 1/2, pp. 57–62.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Leacock, C., Chodorow, M. C-rater: Automated Scoring of Short-Answer Questions. Computers and the Humanities 37, 389–405 (2003). https://doi.org/10.1023/A:1025779619903
Issue Date:
DOI: https://doi.org/10.1023/A:1025779619903