Luca de Alfaro
ABSTRACT
CrowdGrader is a system that lets students submit and collaboratively review and grade homework. We describe the techniques and ideas used in CrowdGrader, and report on the experience of using CrowdGrader in disciplines ranging from Computer Science to Economics, Writing, and Technology. In CrowdGrader, students receive an overall crowd-grade that reflects both the quality of their homework, and the quality of their work as reviewers. This creates an incentive for students to provide accurate grades and helpful reviews of other students' work. Instructors can use the crowd-grades as final grades, or fine-tune the grades according to their wishes. Our results on seven classes show that students actively participate in the grading and write reviews that are generally helpful to the submissions' authors. The results also show that grades computed by CrowdGrader are sufficiently precise to be used as the homework component of class grades. Students report that the main benefits in using CrowdGrader are the quality of the reviews they receive, and the ability to learn from reviewing their peers' work. Instructors can leverage peer learning in their classes, and easily handle homework evaluation in large classes.
- J. Bartholdi, C. Tovey, and M. Trick. Voting schemes for which it can be difficult to tell who won the election. Social Choice and Welfare, 6(2):157--165, 1989.Google Scholar
Cross Ref
- D. J. Boud, R. Cohen, and J. Sampson. Peer learning in higher education: learning from & with each other. Psychology Press, 2001.Google Scholar
- R. Bradley and M. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):pp. 324--345, 1952.Google ScholarCross Ref
- K. Cho, T. Chung, W. King, and C. Schunn. Peer-based computer-supported knowledge refinement: An empirical investigation. Communications of the ACM, 51(3):83--88, 2008. Google ScholarDigital Library
- C. Davidson. How to crowdsource grading, 2009. http://www.hastac.org/blogs/cathy-davidson/how-crowdsource-grading.Google Scholar
- L. de Alfaro and M. Shavlovsky. Crowdgrader: Crowdsourcing the evaluation of homework assignments. Technical Report UCSC-SOE-13-11, UC Santa Cruz, 2013. arXiv:1308.5273.Google Scholar
- J.-C. de Borda. Memoire sur les Elections au Scrutin. 1781.Google Scholar
- C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pages 613--622. ACM, 2001. Google ScholarDigital Library
- A. Elo. The Rating of Chess Players Past and Present. New York, Arco, 1978.Google Scholar
- M. Glickman. Paired Comparison Models with Time-varying Parameters. Harvard University, 1993.Google ScholarCross Ref
- R. Luce. Individual choice behavior : a theoretical analysis. Wiley N.Y, 1959.Google Scholar
- M. Merrifield and D. Saari. Telescope time without tears: A distributed approach to peer review. Astronomy & Geophysics, 50(4):4--16, 2009.Google ScholarCross Ref
- P. Naghizadeh and M. Liu. Incentives, quality, and risk: A look into the NSF proposal review pilot. Arxiv, 1307.6528v1, 2013.Google Scholar
- National Science Foundation. Dear colleague letter: Information to principal investigators (PIs) planning to submit proposals to the sensors and sensing systems (sss) program October 1 , 2013 deadline, 2013.Google Scholar
Index Terms
- CrowdGrader: a tool for crowdsourcing the evaluation of homework assignments
Recommendations
Rubricking Like a Boss: Writing and Using Rubrics For Faster, Fairer Grading of Student Assignments (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science EducationEducators responsible for coordinating a staff to grade a large volume of exams or projects have learned, sometimes the hard way, the importance of writing good rubrics: they keep the graders on track, grading equitably, consistently, and quickly. But ...
Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1As Computer Science Professors, we strive to construct courses that maximally support and contribute to student learning through carefully crafted in-class and out-of-class activities. There is evidence that homework enhances student learning, and that ...
Comments