Dmytro Babik
ABSTRACT
This study examines fidelity of ranking and rating scales in the context of online peer review and assessment. Using the Monte-Carlo simulation technique, we demonstrated that rating scales outperform ranking scales in revealing the relative "true" latent quality of the peer-assessed artifacts via the observed aggregate peer assessment scores. Our analysis focused on a simple, single-round peer assessment process and took into account peer assessment network topology, network size, the number of assessments per artifact, and the correlation statistics used. This methodology allows to separate the effects of structural components of peer assessment from cognitive effects.
- de Alfaro, L. and Shavlovsky, M. 2014. CrowdGrader: A Tool for Crowdsourcing the Evaluation of Homework Assignments. Proceedings of the 45th ACM Technical Symposium on Computer Science Education (New York, NY, USA, 2014), 415--420. Google Scholar
Digital Library
- Alwin, D. and Krosnick, J. 1985. The Measurement of Values in Surveys: A Comparison of Ratings and Rankings. Public Opinion Quarterly. 49, 4 (Dec. 1985), 535--552.Google ScholarCross Ref
- Babik, D., Gehringer, E., Kidd, J., Pramudianto, F. and Tinapple, D. Under Review. Charting the Landscape: A Framework and Systematic Survey of Student Online Peer Review and Assessment Systems. IEEE Transactions on Learning Technologies.Google Scholar
- Babik, D., Gehringer, E., Kidd, J., Pramudianto, F. and Tinapple, D. 2016. Probing the Landscape: Toward a Systematic Taxonomy of Online Peer Assessment Systems in Education. CSPRED 2016: Workshop on Computer-Supported Peer Review in Education.Google Scholar
- Babik, D., Singh, R., Zhao, X. and Ford, E. 2017. What You Think and What I Think: Studying Intersubjectivity in Knowledge artes Evaluation. Information Systems Frontiers. 19, 1, 31--56. Google ScholarDigital Library
- Babik, D., Stevens, S., Waters, A. and Tinapple, D. Under review. The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer Review Networks.Google Scholar
- Bargagliotti, A. and Li, L. 2013. Decision Making Using Rating Systems: When Scale Meets Binary. Decision Sciences. 44, 6, 1121--1137.Google ScholarCross Ref
- Caragiannis, I., Krimpas, G. and Voudouris, A. 2015. Aggregating Partial Rankings with Applications to Peer Grading in Massive Online Open Courses. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 675--683. Google ScholarDigital Library
- Cho, K. and Schunn, C. 2007. Scaffolded Writing and Rewriting in the Discipline: A Web-based Reciprocal Peer Review System. Computers & Education. 48, 3, 409--426. Google ScholarDigital Library
- Craig, B., Busschbach, J. and Salomon, J. 2009. Keep it Simple: Ranking Health States Yields Values Similar to Cardinal Measurement Approaches. Journal of Clinical Epidemiology. 62, 3, 296--305.Google ScholarCross Ref
- Douceur, J.R. 2009. Paper Rating vs. Paper Ranking. ACM SIGOPS Operating Systems Review. 43, 2 (2009), 117--121. Google ScholarDigital Library
- Garlaschelli, D. and Loffredo, M. 2004. Patterns of Link Reciprocity in Directed Networks. Physical Review Letters. 93, 26, 268701.Google ScholarCross Ref
- Gehringer, E., Ehresman, L. and Skrien, D. 2006. Expertiza: Students Helping to Write an OOD Text. Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Systems, Languages, and Applications, 901--906. Google ScholarDigital Library
- Hart-Davidson, W., McLeod, M., Klerkx, C. and Wojcik, M. 2010. A Method for Measuring Helpfulness in Online Peer Review. Proceedings of the 28th ACM International Conference on Design of Communication (New York, NY, USA, 2010), 115--121. Google ScholarDigital Library
- Helbing, D., Yu, W. and Rauhut, H. 2011. Self-Organization and Emergence in Social Systems: Modeling the Coevolution of Social Environments and Cooperative Behavior. The Journal of Mathematical Sociology. 35, 1--3 (Jan. 2011), 177--208.Google ScholarCross Ref
- Joordens, S., Desa, S. and Paré, D. 2009. The Pedagogical Anatomy of Peer Assessment: Dissecting a peerScholar Assignment. Journal of Systemics, Cybernetics & Informatics. 7, 5 (2009).Google Scholar
- Kane, J. and Lawler, E. 1978. Methods of Peer Assessment. Psychological Bulletin. 85, 3 (1978), 555.Google ScholarCross Ref
- Krosnick, J., Thomas, R. and Shaeffer, E. 2003. How Does Ranking Rate?: A Comparison of Ranking and Rating Tasks. Conference Papers - American Association for Public Opinion Research (Annual Meeting, Nashville, TN 2003), N.PAG.Google Scholar
- Landy, F. and Farr, J. 1980. Performance Rating. Psychological Bulletin. 87, 1 (1980), 72--107.Google ScholarCross Ref
- Love, K. 1981. Comparison of Peer Assessment Methods: Reliability, Validity, Friendship Bias, and User Reaction. Journal of Applied Psychology. 66, 4 (1981), 451--457.Google ScholarCross Ref
- Luce, R. and Perry, A. 1949. A Method of Matrix Analysis of Group Structure. Psychometrika. 14, 2 (Jun. 1949), 95--116.Google ScholarCross Ref
- Murphy, K., Cleveland, J., Skattebo, A. and Kinney, T. 2004. Raters Who Pursue Different Goals Give Different Ratings. Journal of Applied Psychology. 89, 1 (2004), 158--164.Google ScholarCross Ref
- Opsahl, T. and Panzarasa, P. 2009. Clustering in Weighted Networks. Social Networks. 31, 2 (May 2009), 155--163.Google ScholarCross Ref
- Raman, K. and Joachims, T. 2014. Methods for Ordinal Peer Grading. arXiv: 1404.3656 {cs}. (Apr. 2014). Google ScholarDigital Library
- Rankin, W. and Grube, J. 1980. A Comparison of Ranking and Rating Procedures for Value System Measurement. European Journal of Social Psychology. 10, 3 (1980), 233--246.Google ScholarCross Ref
- Russell, A. 2001. Calibrated Peer Review: A Writing and Critical-Thinking Instructional Tool. UCLA, Chemistry. 2001, (2001).Google Scholar
- Shah, N., Bradley, J., Parekh, A., Wainwright, M. and Ramchandran, K. 2013. A Case for Ordinal Peer Evaluation in MOOCs. NIPS Workshop on Data Driven Education. (Dec. 2013).Google Scholar
- Shannon, C. and Weaver, W. 1964. The Mathematical Theory of Communication. University of Illinois Press. Google ScholarDigital Library
- Simon, H. 1991. Bounded Rationality and Organizational Learning. Organization Science. 2, 1 (Feb. 1991), 125--134. Google ScholarDigital Library
- Song, Y., Guo, Y. and Gehringer, E. 2017. An Exploratory Study of Reliability of Ranking vs. Rating in Peer Assessment. International Journal of Educational and Pedagogical Sciences. 11, 10 (2017), 5.Google Scholar
- Stevens, S., Waters, A., Babik, D. and Tinapple, D. 2016. Efficacy of Peer Review Network Structures: The Effects of Reciprocity and Clustering. ICIS 2016 Proceedings. (Dec. 2016).Google Scholar
- Tinapple, D., Olson, L. and Sadauskas, J. 2013. CritViz: Web-Based Software Supporting Peer Critique in Large Creative Classrooms. Bulletin of the IEEE Technical Committee on Learning Technology. 15, 1 (2013), 29.Google Scholar
- Topping, K. 2009. Peer Assessment. Theory into Practice. 48, 1 (2009), 20--27.Google ScholarCross Ref
- Topping, K. 1998. Peer Assessment Between Students in Colleges and Universities. Review of Educational Research. 68, 3, 249--276.Google ScholarCross Ref
- Topping, K. 2005. Trends in Peer Learning. Educational Psychology. 25, 6 (Dec. 2005), 631--645.Google ScholarCross Ref
- Traulsen, A., Hauert, C., Silva, H., Nowak, M. and Sigmund, K. 2009. Exploration Dynamics in Evolutionary Games. Proceedings of the National Academy of Sciences. 106, 3 (Jan. 2009), 709--712.Google ScholarCross Ref
- Waters, A., Tinapple, D. and Baraniuk, R. 2015. BayesRank: A Bayesian Approach to Ranked Peer Grading. ACM Conference on Learning at Scale (Vancouver, Mar. 2015). Google ScholarDigital Library
Index Terms
- Comparison of Ranking and Rating Scales in Online Peer Assessment: Simulation Approach
Recommendations
Does evaluating peer assessment accuracy and taking it into account in calculating assessor’s final score enhance online peer assessment quality?
AbstractPeer assessment has an important role in teaching and learning nowadays. However, the existing techniques tend to be limited due to a lack of a suitable means to evaluate the accuracy of peer assessment activities and supervise peer assessment ...
Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1As Computer Science Professors, we strive to construct courses that maximally support and contribute to student learning through carefully crafted in-class and out-of-class activities. There is evidence that homework enhances student learning, and that ...
Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives
L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ ScaleMassive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer ...
Comments