skip to main content
10.1145/3430665.3456338acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

Investigating Accuracy and Perceived Value of Feedback in Peer Code Review Using Gamification

Published: 26 June 2021 Publication History

Abstract

The practice of peer code review has been shown to deliver a variety of benefits to programming students. These include learning from producing and receiving feedback, and from being exposed to a range of problem-solving approaches and solutions. However, the success of a peer code review activity depends on the quality and accuracy of the reviews that students produce, and prior work has shown that these can sometimes be poor. One approach for addressing this problem is to incorporate motivational incentives directly into the design of the code review platform. In this research, we explore the use of gamification in an online peer code review tool, where game-like elements are used to reward students for generating accurate and helpful reviews. We report the results of a randomized controlled study (n=171) that measures both review accuracy and the perceived value of the feedback produced. Although quantitative ratings of the review quality did not differ significantly between control and experimental conditions, we observed interesting trends relating to the perceived value of the feedback. Students in both groups had similar views regarding the usefulness of the feedback they received on their own work, however students in the experimental condition tended to express more positive sentiments towards the quality of the feedback they produced for their peers and observed from other reviewers.

References

[1]
Fernando Almeida. 2018. Framework for software code reviews and inspections in a classroom environment. Int. J. Modern Educ. Comput. Sci.(IJMECS)10, 10(2018), 31--39.
[2]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2006), 77--101. Issue 2. https://doi.org/10.1191/1478088706qp063oa
[3]
Antonella Carbonaro and Mirko Ravaioli. 2017. Peer assessment to promote Deep Learning and to reduce a Gender Gap in the Traditional Introductory Programming Course.Journal of e-Learning and Knowledge Society 13, 3 (September 2017). https://www.learntechlib.org/p/180982
[4]
Simone de Sousa Borges, Vinicius H. S. Durelli, Helena Macedo Reis, and Seiji Isotani. 2014. A systematic mapping on gamification applied to education. In Proceedings of the 29th Annual ACM Symposium on Applied Computing - SAC '14. 216--222. https://doi.org/10.1145/2554850.2554956
[5]
Stavros Demetriadis, Tina Egerter, Frank Hanisch, and Frank Fischer. 2011. Peer review-based scripted collaboration to support domain-specific and domain-general knowledge acquisition in computer science. Computer Science Education 21, 1 (2011), 29--56. https://doi.org/10.1080/08993408.2010.539069 arXiv: https://doi.org/10.1080/08993408.2010.539069
[6]
Paul Denny. 2013. The Effect of Virtual Achievements on Student Engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 763--772. https://doi.org/10.1145/2470654.2470763
[7]
Paul Denny, Fiona McDonald, Ruth Empson, Philip Kelly, and Andrew Petersen. 2018. Empirical Support for a Causal Relationship Between Gamification and Learning Outcomes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3173574.3173885
[8]
Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O'Hara, and Dan Dixon. 2011. Gamification. Using Game-design Elements in Non-gaming Contexts. In CHI '11 Extended Abstracts on Human Factors in Computing Systems(Vancouver, BC, Canada) (CHI EA '11). ACM, New York, NY, USA, 2425--2428. https://doi.org/10.1145/1979742.1979575
[9]
Lu Ding, Erkan Er, and Michael Orey. 2018. An exploratory study of student engagement in gamified online discussions.Computers & Education 120 (2018), 213 -- 226. https://doi.org/10.1016/j.compedu.2018.02.007
[10]
J. Hamari, J. Koivisto, and H. Sarsa. 2014. Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification. In 2014 47th Hawaii International Conference on System Sciences. 3025--3034. https://doi.org/10.1109/HICSS.2014.377
[11]
John Hamer, Kenneth T. K. Ma, and Hugh H. F. Kwong. 2005. A Method of Automatic Grade Calibration in Peer Assessment. In Proceedings of the 7th Australasian Conference on Computing Education - Volume 42(Newcastle, New South Wales, Australia) (ACE '05). Australian Computer Society, Inc., AUS, 67--72.
[12]
John Hamer, Helen Purchase, Andrew Luxton-Reilly, and Paul Denny. 2015. A comparison of peer and tutor feedback. Assessment & Evaluation in Higher Education 40, 1 (2015), 151--164. https://doi.org/10.1080/02602938.2014.893418 arXiv: https://doi.org/10.1080/02602938.2014.893418
[13]
John Hamer, Helen C. Purchase, Paul Denny, and Andrew Luxton-Reilly. 2009.Quality of Peer Assessment in CS1. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop(Berkeley, CA, USA) (ICER'09). Association for Computing Machinery, New York, NY, USA, 27--36. https://doi.org/10.1145/1584322.1584327
[14]
Francis Heylighen, Iavor Kostov, and Mixel Kiemen. 2013. Mobilization Systems:technologies for motivating and coordinating human action. The New Development Paradigm: Education, Knowledge Economy and Digital Futures. Routledge. Retrieved from http://pcp.vub.ac.be/Papers/MobilizationSystems.pdf(2013).
[15]
Chih-Chao Hsu and Tzone-I. Wang. 2018. Applying game mechanics and student-generated questions to an online puzzle-based game learning system to promote algorithmic thinking skills. Computers & Education 121 (2018), 73 -- 88. https://doi.org/10.1016/j.compedu.2018.02.002
[16]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2020. Gamification of student peer review in education: A systematic literature review. Education and Information Technologies 25, 6 (01 Nov 2020), 5205--5234. https://doi.org/10.1007/s10639-020--10228-x
[17]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2020. A Review of Peer Code Review in Higher Education. ACM Trans. Comput. Educ. 20, 3, Article 22 (Sept. 2020), 25 pages. https://doi.org/10.1145/3403935
[18]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2021. Improving Student Peer Code Review Using Gamification. In Australasian Computing Education Conference(Virtual, SA, Australia)(ACE '21). Association for Computing Machinery, New York, NY, USA, 80--87. https://doi.org/10.1145/3441636.3442308
[19]
Stephan Krusche, Mjellma Berisha, and Bernd Bruegge. 2016. Teaching Code Review Management Using Branch Based Workflows. In Proceedings of the 38th International Conference on Software Engineering Companion(Austin, Texas) (ICSE'16). ACM, New York, NY, USA, 384--393. https://doi.org/10.1145/2889160.2889191
[20]
Z. Kubincová and M. Homola. 2017. Code review in computer science courses:Take one. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)10473 LNCS (2017), 125--135. https://doi.org/10.1007/978--3--319--66733--1_14
[21]
Hongli Li, Yao Xiong, Xiaojiao Zang, Mindy L. Kornhaber, Youngsun Lyu, Kyung Sun Chung, and Hoi K. Suen. 2016. Peer assessment in the digital age: a meta-analysis comparing peer and teacher ratings. Assessment & Evaluation in Higher Education 41, 2 (2016), 245--264. https://doi.org/10.1080/02602938.2014.999746 arXiv: https://doi.org/10.1080/02602938.2014.999746
[22]
Heng Luo, Anthony Robinson, and Jae-Young Park. 2014. Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects. Online Learning Journal 18, 2(June 2014). https://www.learntechlib.org/p/183756
[23]
Andrew Luxton-Reilly. 2009. A systematic review of tools that support peer assessment. Computer Science Education 19, 4 (2009), 209--232. https://doi.org/10.1080/08993400903384844 arXiv: https://doi.org/10.1080/08993400903384844
[24]
Jenni Majuri, Jonna Koivisto, and Juho Hamari. 2018. Gamification of education and learning: A review of empirical literature. In GamiFIN Conference 2018, Pori, Finland. 11--19.
[25]
Gabriela Morales-Martinez, Paul Latreille, and Paul Denny. 2020. Nationality and Gender Biases in Multicultural Online Learning Environments: The Effects of Anonymity. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376283
[26]
Heather Pon-Barry, Becky Wai-Ling Packard, and Audrey St. John. 2017. Expanding capacity and promoting inclusion in introductory computer science: a focus on near-peer mentor preparation and code review. Computer Science Education 27, 1 (2017), 54--77. https://doi.org/10.1080/08993408.2017.1333270 arXiv: https://doi.org/10.1080/08993408.2017.1333270
[27]
Mitchell Rogers, Wendy Yao, Andrew Luxton-Reilly, Juho Leinonen, Danielle Lottridge, and Paul Denny. 2021. Exploring Personalization of Gamification in an Introductory Programming Course. Association for Computing Machinery, New York, NY, USA, 1121--1127. https://doi.org/10.1145/3408877.3432402
[28]
Xiangyu Song, Seth Copen Goldstein, and Majd Sakr. 2020. Using Peer Code Review as an Educational Tool. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education(Trondheim, Norway) (ITiCSE '20). Association for Computing Machinery, New York, NY, USA, 173--179. https://doi.org/10.1145/3341525.3387370
[29]
Martijn Stegeman, Erik Barendsen, and Sjaak Smetsers. 2016. Designing a Rubric for Feedback on Code Quality in Programming Courses. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research(Koli, Finland) (Koli Calling '16). Association for Computing Machinery, New York, NY, USA, 160--164. https://doi.org/10.1145/2999541.2999555
[30]
Gustavo F. Tondello and Lennart E. Nacke. 2020. Validation of User Preferences and Effects of Personalized Gamification on Task Performance. Frontiers in Computer Science 2 (2020), 1--23. https://doi.org/10.3389/fcomp.2020.00029
[31]
Deborah A. Trytten. 2005. A Design for Team Peer Code Review. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education(St. Louis, Missouri, USA) (SIGCSE '05). ACM, New York, NY, USA, 455--459. https://doi.org/10.1145/1047344.1047492
[32]
Y. Wang, H. Li, Y. Feng, Y. Jiang, and Y. Liu. 2012. Assessment of programming language learning based on peer code review model: Implementation and experience report. Computers & Education 59, 2 (2012), 412--422. https://doi.org/10.1016/j.compedu.2012.01.007 cited By 43.
[33]
Kevin Werbach and Dan Hunter. 2012. For the win: How game thinking can revolutionize your business. Wharton digital press.
[34]
Eric Zimmerling, Christoph E. Hollig, Philipp G. Sandner, and Isabell M. Welpe. 2019. Exploring the influence of common game elements on ideation output and motivation. Journal of Business Research94 (2019), 302 -- 312. https://doi.org/10.1016/j.jbusres.2018.02.030

Cited By

View all
  • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
  • (2023)A Peer Review Approach to Grading Projects in Computer Courses2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10343176(1-7)Online publication date: 18-Oct-2023
  • (2023)Lessons Learned From Four Computing Education Crowdsourcing SystemsIEEE Access10.1109/ACCESS.2023.325364211(22982-22992)Online publication date: 2023
  • Show More Cited By

Index Terms

  1. Investigating Accuracy and Perceived Value of Feedback in Peer Code Review Using Gamification

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1
    June 2021
    611 pages
    ISBN:9781450382144
    DOI:10.1145/3430665
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 June 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. CS1
    2. code review
    3. computing education
    4. gamification
    5. higher education
    6. peer code review
    7. peer review
    8. programming course

    Qualifiers

    • Research-article

    Funding Sources

    • LPDP (Indonesian Endowment Fund for Education)

    Conference

    ITiCSE 2021
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 552 of 1,613 submissions, 34%

    Upcoming Conference

    ITiCSE '25
    Innovation and Technology in Computer Science Education
    June 27 - July 2, 2025
    Nijmegen , Netherlands

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)55
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 23 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
    • (2023)A Peer Review Approach to Grading Projects in Computer Courses2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10343176(1-7)Online publication date: 18-Oct-2023
    • (2023)Lessons Learned From Four Computing Education Crowdsourcing SystemsIEEE Access10.1109/ACCESS.2023.325364211(22982-22992)Online publication date: 2023
    • (2023)Computing Education Research in AustralasiaPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_17(373-394)Online publication date: 5-Jan-2023
    • (2022)Teaching effective Cybersecurity through escape the classroom paradigm2022 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON52537.2022.9766684(17-23)Online publication date: 28-Mar-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media