skip to main content
10.1145/2556325.2566243acmconferencesArticle/Chapter ViewAbstractPublication Pagesl-at-sConference Proceedingsconference-collections
research-article

Divide and correct: using clusters to grade short answers at scale

Published: 04 March 2014 Publication History

Abstract

In comparison to multiple choice or other recognition-oriented forms of assessment, short answer questions have been shown to offer greater value for both students and teachers; for students they can improve retention of knowledge, while for teachers they provide more insight into student understanding. Unfortunately, the same open-ended nature which makes them so valuable also makes them more difficult to grade at scale. To address this, we propose a cluster-based interface that allows teachers to read, grade, and provide feedback on large groups of answers at once. We evaluated this interface against an unclustered baseline in a within-subjects study with 25 teachers, and found that the clustered interface allows teachers to grade substantially faster, to give more feedback to students, and to develop a high-level view of students' understanding and misconceptions.

References

[1]
Anderson, R. and Biddle, W. On asking people questions about what they are reading. Psychology of learning and motivation 9, (1975).
[2]
Basu, S., Jacobs, C., and Vanderwende, L. Powergrading: a Clustering Approach to Amplify Human Effort for Short Answer Grading. TACL 1, (2013), 391--402.
[3]
Bloom, B. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Edu. Researcher 13, 6 (1984), 4--16.
[4]
Brookhart, S. Teachers' grading: Practice and theory. Applied Measurement in Edu., (1994).
[5]
Cross, L. and Frary, R. Hodgepodge grading: Endorsed by students and teachers alike. Applied Measurement in Edu., October 2013 (1999), 37--41.
[6]
Hearst, M. The debate on automated essay grading. Intelligent Systems and their Applications, (2000).
[7]
Heywood, J. Assessment in higher education: Student learning, teaching, programmes and institutions. 2000.
[8]
Jordan, S. and Mitchell, T. e-Assessment for learning? The potential of short-answer free-text questions with tailored feedback. British Journal of Edu. Tech. 40, 2 (2009), 371--385.
[9]
Karpicke, J.D. and Roediger, H.L. The critical importance of retrieval for learning. Science 319, 5865 (2008), 966--8.
[10]
Markoff, J. Essay-Grading Software Offers Professors a Break. The New York Times, 2013.
[11]
McMillan, J. Secondary teachers' classroom assessment and grading practices. Edu. Measurement: Issues and Practice, (2001).
[12]
Mohler, M.A.G., Bunescu, R., and Mihalcea, R. Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments. Proc. ACL, (2011).
[13]
Mory, E. Feedback research revisited. In D.J. Mahwah, ed., Handbook of Research on Educational Communications and Technology. 2004, 745--784.
[14]
Perelman, L. Critique (Ver. 3.4) of Mark D. Shermis & Ben Hammer, "Contrasting State-of-the-Art Automated Scoring of Essays: Analysis."2013.
[15]
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., and Koller, D. Tuned Models of Peer Assessment in MOOCs. Proc. EDM, (2013).
[16]
Poulos, A. and Mahony, M.J. Effectiveness of feedback: the students' perspective. Assessment & Evaluation in Higher Edu. 33, 2 (2008), 143--154.
[17]
Reily, K., Finnerty, P.L., and Terveen, L. Two peers are better than one: aggregating peer reviews for computing assignments is surprisingly accurate. Proc. GROUP, ACM Press (2009), 115.
[18]
Sadler, P. and Good, E. The Impact of Self- and Peer-Grading on Student Learning. Edu. Assessment 11, 1 (2006), 1--31.
[19]
Scriven, M. The methodology of evaluation. In R.E. Stake, ed., AERSA Monograph Series on Curriculum Evaluation. Rand McNally, Chicago, 1967.
[20]
Thorpe, M. Assessment and "third generation" distance education. Distance Edu. 19, 2 (1998), 265--286.
[21]
Weld, D., Adar, E., and Chilton, L. Personalized Online Education--A Crowdsourcing Challenge. Proc. AAAI, workshop on Human Computation, (2012), 159--163.

Cited By

View all
  • (2024)Propagating Large Language Models Programming FeedbackProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664665(366-370)Online publication date: 9-Jul-2024
  • (2024)Automation and Assessment: Exploring Ethical Issues of Automated Grading Systems from a Relational Ethics ApproachFraming Futures in Postdigital Education10.1007/978-3-031-58622-4_12(209-226)Online publication date: 10-Jul-2024
  • (2023)The Student Zipf Theory: Inferring Latent Structures in Open-Ended Student Work To Help EducatorsLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576116(464-475)Online publication date: 13-Mar-2023
  • Show More Cited By

Index Terms

  1. Divide and correct: using clusters to grade short answers at scale

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    L@S '14: Proceedings of the first ACM conference on Learning @ scale conference
    March 2014
    234 pages
    ISBN:9781450326698
    DOI:10.1145/2556325
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 March 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. assessment
    2. clustering
    3. clustering interfaces
    4. grading
    5. grading interfaces
    6. moocs
    7. user interfaces

    Qualifiers

    • Research-article

    Conference

    L@S 2014
    Sponsor:
    L@S 2014: First (2014) ACM Conference on Learning @ Scale
    March 4 - 5, 2014
    Georgia, Atlanta, USA

    Acceptance Rates

    L@S '14 Paper Acceptance Rate 14 of 38 submissions, 37%;
    Overall Acceptance Rate 117 of 440 submissions, 27%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)29
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Propagating Large Language Models Programming FeedbackProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664665(366-370)Online publication date: 9-Jul-2024
    • (2024)Automation and Assessment: Exploring Ethical Issues of Automated Grading Systems from a Relational Ethics ApproachFraming Futures in Postdigital Education10.1007/978-3-031-58622-4_12(209-226)Online publication date: 10-Jul-2024
    • (2023)The Student Zipf Theory: Inferring Latent Structures in Open-Ended Student Work To Help EducatorsLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576116(464-475)Online publication date: 13-Mar-2023
    • (2023)Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematicsJournal of Computer Assisted Learning10.1111/jcal.1279339:3(823-840)Online publication date: 13-Feb-2023
    • (2023)Reducing Workload in Short Answer Grading Using Machine LearningInternational Journal of Artificial Intelligence in Education10.1007/s40593-022-00322-134:2(247-273)Online publication date: 28-Feb-2023
    • (2022)Clustering students’ writing behaviors using keystroke logging: a learning analytic approach in EFL writingLanguage Testing in Asia10.1186/s40468-021-00150-512:1Online publication date: 7-Feb-2022
    • (2022)Evaluating CodeClusters for Effectively Providing Feedback on Code Submissions2022 IEEE Frontiers in Education Conference (FIE)10.1109/FIE56618.2022.9962751(1-9)Online publication date: 8-Oct-2022
    • (2022)An Interactive Short Answer Grading System Based on Active Learning Models2022 4th International Conference on Computer Science and Technologies in Education (CSTE)10.1109/CSTE55932.2022.00065(314-321)Online publication date: May-2022
    • (2022)Automated short answer grading with computer-assisted grading example acquisition based on active learningInteractive Learning Environments10.1080/10494820.2022.2137530(1-18)Online publication date: 5-Dec-2022
    • (2021)Clustering of Handwritten Mathematical Expressions for Computer-Assisted MarkingIEICE Transactions on Information and Systems10.1587/transinf.2020EDP7087E104.D:2(275-284)Online publication date: 1-Feb-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media