skip to main content
10.1145/3507923.3507955acmotherconferencesArticle/Chapter ViewAbstractPublication PagescsercConference Proceedingsconference-collections
research-article
Open access

Insights from Peer Reviewing in Large University Courses

Published: 13 April 2022 Publication History

Abstract

Teaching a mandatory course for undergraduate computer science students with up to 750 students per semester, we have been making extensive use of peer reviewing. During the semester, each student has to work on a set of assignments. After finishing an assignment, the student has to write three peer reviews for three pieces of work by different, anonymous peers. One of the problems in the use of (student) peer reviewing in large university courses is the quality of written reviews. To address this problem, we devised various provisions to maintain or increase reviewing quality. In this article, we describe one of these provisions, namely the use of three different types of reviews instead of using identical review types three times in a row: guided reviews, open reviews, checkbox reviews. Our aim in this article is to research the impact of these different review types on the students’ experience and acceptance of the reviewing process to inform the design of a reviewing process that better fits the students’ needs. To gain such insights, we gathered feedback using a survey, which was completed by 101 students. Using qualitative analysis, we extracted and defined room for improvement and discuss possible changes for our current peer review system and process. Our learnings show insights into the types of reviews students prefer, and hint at some advantages and pitfalls of peer reviewing that can have substantial impact on the design and application of such a system in large university courses.

References

[1]
M Al-Smadi, C Gütl, and F Kappe. 2010. Towards an Enhanced Approach for Peer-Assessment Activities. In Advanced Learning Technologies (ICALT), 2010 IEEE 10th International Conference on. 637–641. https://doi.org/10.1109/ICALT.2010.181
[2]
Stephen P. Balfour. 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review (TM). Research & Practice in Assessment 8 (2013). http://search.proquest.com/openview/429a5ba1673266b30d4ebc7d9fb3fc69/1?pq-origsite=gscholar
[3]
B Basnet, L Brodie, and J Worden. 2010. Peer assessment of assignment. In Frontiers in Education Conference (FIE), 2010 IEEE. T1G–1–T1G–2. https://doi.org/10.1109/FIE.2010.5673273
[4]
Christine Bauer, Kathrin Figl, Michael Derntl, Peter Paul Beran, and Sonja Kabicher. 2009. The Student View on Online Peer Reviews. Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education. 41, 3 (Jul 2009), 26–30. https://doi.org/10.1145/1595496.1562892
[5]
David E. Berry and Kelli L. Fawkes. 2010. Constructing the components of a lab report using peer review. Journal of Chemical Education 87, 1 (2010), 57–61.
[6]
Matthew J. Bietz. 2008. Effects of communication media on the interpretation of critical feedback. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. ACM, 467–476. http://dl.acm.org/citation.cfm?id=1460637
[7]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
[8]
George Brown. 2001. Assessment: A guide for lecturers. Vol. 3. LTSN Generic Centre York.
[9]
Julia Cambre, Scott Klemmer, and Chinmay Kulkarni. 2018. Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, 1–13. https://doi.org/10.1145/3173574.3173868
[10]
Kwangsu Cho and Charles MacArthur. 2010. Student revision with peer and expert reviewing. Learning and Instruction 20, 4 (2010), 328–338. https://doi.org/10.1016/j.learninstruc.2009.08.006
[11]
Carleton Coffrin, Linda Corrin, Paula de Barba, and Gregor Kennedy. 2014. Visualizing patterns of student engagement and performance in MOOCs. ACM Press, 83–92. https://doi.org/10.1145/2567574.2567586
[12]
Raquel M Crespo García, Abelardo Pardo, and Carlos Delgado Kloos. 2004. An adaptive strategy for peer review. In Frontiers in Education Conference. Ieee, F3F–7–13 Vol. 2. https://doi.org/10.1109/FIE.2004.1408636
[13]
Christopher Frauenberger and Peter Purgathofer. 2019. Ways of thinking in informatics. Commun. ACM 62, 7 (Jun 2019), 58–64. https://doi.org/10.1145/3329674
[14]
Vahid Garousi. 2010. Applying peer reviews in software engineering education: An experiment and lessons learned. IEEE Transactions on Education 53, 2 (2010), 182–193.
[15]
J. Hattie and H. Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1 (Mar 2007), 81–112. https://doi.org/10.3102/003465430298487
[16]
Thanos Hatziapostolou and Iraklis Paraskakis. 2010. Enhancing the Impact of Formative Feedback on Student Learning through an Online Feedback System.Electronic Journal of e-Learning 8, 2 (2010), 111–122.
[17]
Amy J Henley and Florence D DiGennaro Reed. 2015. Should you order the feedback sandwich? Efficacy of feedback sequence and timing. Journal of Organizational Behavior Management 35, 3-4(2015), 321–335.
[18]
Catherine M. Hicks, Vineet Pandey, C. Ailie Fraser, and Scott Klemmer. 2016. Framing Feedback: Choosing Review Environment Features that Support High Quality Peer Assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM Press, 458–469. https://doi.org/10.1145/2858036.2858195
[19]
David A. Joyner, Angela Smiley, Amy Bruckman, Ashok Goel, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langson, Isabel Lupiani, Mike Lustig, and et al.2016. Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. ACM Press, 399–408. https://doi.org/10.1145/2876034.2876044
[20]
Rachel Jug, Xiaoyin “Sara” Jiang, and Sarah M Bean. 2019. Giving and receiving effective feedback: A review article and how-to guide. Archives of pathology & laboratory medicine 143, 2 (2019), 244–250.
[21]
Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction 20, 6 (Dec 2013), 1–31. https://doi.org/10.1145/2505057
[22]
Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale. ACM Press, 75–84. https://doi.org/10.1145/2724660.2724670
[23]
Chinmay E. Kulkarni, Richard Socher, Michael S. Bernstein, and Scott R. Klemmer. 2014. Scaling short-answer grading by combining peer assessment with algorithmic scoring. In Proceedings of the first ACM conference on Learning @ scale conference - L@S ’14. ACM Press, 99–108. https://doi.org/10.1145/2556325.2566238
[24]
E.Z.-F. Liu, S S J Lin, Chi-Huang Chiu, and Shyan-Ming Yuan. 2001. Web-based peer review: the learner as both adapter and reviewer. Education, IEEE Transactions on 44, 3 (2001), 246–251. https://doi.org/10.1109/13.940995
[25]
Yanxin Lu, Joe Warren, Christopher Jermaine, Swarat Chaudhuri, and Scott Rixner. 2015. Grading the Graders: Motivating Peer Graders in a MOOC. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 680–690. https://doi.org/10.1145/2736277.2741649
[26]
Naemi Luckner and Peter Purgathofer. 2015. Exploring the Use of Peer Review in Large University Courses. Interaction Design and Architecture(s) Journal25 (2015), 21–38.
[27]
Naemi Luckner, Peter Purgathofer, and Geraldine Fitzpatrick. 2018. Increasing Peer Review Quality in Online Learning Systems. In Proceedings of the 4th International Conference on Higher Education Advances. Editorial Universitat Politècnica de València, 321–328.
[28]
Naemi Luckner, Peter Purgathofer, and Geraldine Fitzpatrick. 2019. Involving Students in the Co-Creation of a Complex, Evolving Learning Environment. Interaction Design and Architecture(s) Journal 42 (2019), 70–92.
[29]
Christian Lundquist, Martin a. Skoglund, Karl Granström, and Torkel Glad. 2013. Insights from implementing a system for peer review. IEEE Transactions on Education 56, 3 (2013), 261–267. https://doi.org/10.1109/TE.2012.2211876
[30]
Elizabeth Molloy, Rola Ajjawi, Margaret Bearman, Christy Noble, Joy Rudland, and Anna Ryan. 2020. Challenging feedback myths: values, learner involvement and promoting effects beyond the immediate task. Medical education 54, 1 (2020), 33–39.
[31]
Alistair Mutch. 2003. Exploring the practice of feedback to students. Active learning in higher education 4, 1 (2003), 24–38.
[32]
Lynette Nagel and Theuns G Kotzé. 2010. Supersizing e-learning: What a CoI survey reveals about teaching presence in a large online class. The Internet and Higher Education 13, 1–2 (2010), 45–51. https://doi.org/10.1016/j.iheduc.2009.12.001
[33]
Tricia J. Ngoon, C. Ailie Fraser, Ariel S. Weingarten, Mira Dontcheva, and Scott Klemmer. 2018. Interactive Guidance Techniques for Improving Creative Feedback. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, 1–11. https://doi.org/10.1145/3173574.3173629
[34]
Jay Parkes, Sara Abercrombie, and Teresita McCarty. 2013. Feedback sandwiches affect perceptions but not performance. Advances in Health Sciences Education 18, 3 (2013), 397–407.
[35]
Jon Pearce, Raoul Mulder, Chi Baik, University of Melbourne, and Centre for the Study of Higher Education. 2009. Involving students in peer review: case studies and practical strategies for university teaching. Centre for the Study of Higher Education, University of Melbourne.
[36]
Nancy J Pelaez. 2002. Problem-based writing with peer review improves academic performance in physiology.Advances in physiology education 26, 1–4 (2002), 174–184.
[37]
Jakub Procházka, Martin Ovcari, and Michal Durinik. 2020. Sandwich feedback: The empirical evidence of its effectiveness. Learning and Motivation 71 (2020), 101649.
[38]
Peter Purgathofer and Wilfried Reinthaler. 2008. Exploring the “Massive Multiplayer E-Learning” Concept. Ed-Media Invited TalkPurgathofer 2005 (2008), 1–9.
[39]
Horst WJ Rittel and Melvin M Webber. 1973. Dilemmas in a general theory of planning. Policy sciences 4, 2 (1973), 155–169.
[40]
Amber Settle, Charles Wilcox, and Chad Settle. 2011. Engaging Game Design Students Using Peer Evaluation. In Proceedings of the 2011 Conference on Information Technology Education(SIGITE ’11). ACM, 73–78. https://doi.org/10.1145/2047594.2047614
[41]
Scott Turner, Manuel A Pérez-Quiñones, Stephen Edwards, and Joseph Chase. 2011. Student Attitudes and Motivation for Peer Review in CS2. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education(SIGCSE ’11). ACM, 347–352. https://doi.org/10.1145/1953163.1953268
[42]
Marjo van Zundert, Dominique Sluijsmans, and Jeroen van Merriënboer. 2010. Effective peer assessment processes: Research findings and future directions. Learning and Instruction 20, 4 (2010), 270–279. https://doi.org/10.1016/j.learninstruc.2009.08.004
[43]
CW Von Bergen, Martin S Bressler, and Kitty Campbell. 2014. The sandwich feedback method: not very tasty. Journal of Behavioral Studies in business 7 (2014).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
CSERC '21: Proceedings of the 10th Computer Science Education Research Conference
November 2021
112 pages
ISBN:9781450385763
DOI:10.1145/3507923
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 April 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computer science education
  2. peer feedback
  3. peer review
  4. technology-enhanced learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

CSERC '21
CSERC '21: The 10th Computer Science Education Research Conference
November 22 - 23, 2021
Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 24 of 60 submissions, 40%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 234
    Total Downloads
  • Downloads (Last 12 months)136
  • Downloads (Last 6 weeks)22
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media