skip to main content
10.1145/3441636.3442308acmotherconferencesArticle/Chapter ViewAbstractPublication Pagesaus-ceConference Proceedingsconference-collections
research-article

Improving Student Peer Code Review Using Gamification

Published: 17 March 2021 Publication History

Abstract

Peer code review has been shown to have several benefits for students, including the development of both technical skills and soft skills. However, a lack of motivation has been identified as one of the barriers to successful peer code review in programming courses. Low motivation may result in students avoiding or delaying their peer review tasks, reducing the potential benefits. In this study, gamification is used to overcome this barrier. We focus on motivating two behaviors: increasing the number of reviews submitted by students, and encouraging students to submit those reviews early. We conduct a randomized controlled study (N = 178) that compares the behavior of a control group engaged in peer code review using an online tool, with a gamification group that uses a modified version of the tool that includes targeted game elements. The results show a statistically significant difference in the number of submitted reviews between the control and gamification groups. Furthermore, the majority of students in the gamification group report that the game elements motivate them. Based on our findings, the game elements and game mechanics seem to be a promising method to motivate students in online peer code review activities.

References

[1]
Deepika Badampudi, Ricardo Britto, and Michael Unterkalmsteiner. 2019. Modern Code Reviews - Preliminary Results of a Systematic Mapping Study. In Proc. of the Evaluation and Assessment on Soft. Eng. (Copenhagen, Denmark) (EASE ’19). ACM, New York, NY, USA, 340–345. https://doi.org/10.1145/3319008.3319354
[2]
Roy Ballantyne, Karen Hughes, and Aliisa Mylonas. 2002. Developing Procedures for Implementing Peer Assessment in Large Classes Using an Action Research Process. Assessment & Evaluation in Higher Education 27, 5 (2002), 427–441. https://doi.org/10.1080/0260293022000009302
[3]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
[4]
Antonella Carbonaro and Mirko Ravaioli. 2017. Peer assessment to promote Deep Learning and to reduce a Gender Gap in the Traditional Introductory Programming Course. Journal of e-Learning and Knowledge Society 13, 3 (September 2017). https://www.learntechlib.org/p/180982
[5]
Simone de Sousa Borges, Vinicius H. S. Durelli, Helena Macedo Reis, and Seiji Isotani. 2014. A systematic mapping on gamification applied to education. In Proceedings of the 29th Annual ACM Symposium on Applied Computing - SAC ’14. 216–222. https://doi.org/10.1145/2554850.2554956
[6]
Paul Denny. 2013. The Effect of Virtual Achievements on Student Engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). ACM, New York, NY, USA, 763–772. https://doi.org/10.1145/2470654.2470763
[7]
Paul Denny, Fiona McDonald, Ruth Empson, Philip Kelly, and Andrew Petersen. 2018. Empirical Support for a Causal Relationship Between Gamification and Learning Outcomes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173885
[8]
Paul Denny, Jacqueline Whalley, and Juho Leinonen. 2021. Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback. In Australasian Computing Education Conference(Virtual, SA, Australia) (ACE’21). ACM, New York, NY, USA. https://doi.org/10.1145/3441636.3442309
[9]
Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O’Hara, and Dan Dixon. 2011. Gamification. Using Game-design Elements in Non-gaming Contexts. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA ’11). ACM, New York, NY, USA, 2425–2428. https://doi.org/10.1145/1979742.1979575
[10]
Lu Ding, Erkan Er, and Michael Orey. 2018. An exploratory study of student engagement in gamified online discussions. Computers & Education 120 (2018), 213 – 226. https://doi.org/10.1016/j.compedu.2018.02.007
[11]
K Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. 1993. The role of deliberate practice in the acquisition of expert performance.Psychological Review 100, 3 (1993), 363–406. https://doi.org/10.1037/0033-295X.100.3.363
[12]
Edward F. Gehringer, Donald D. Chinn, Manuel A. Pérez-Quiñones, and Mark A. Ardis. 2005. Using Peer Review in Teaching Computing. In Proc. of the 36th SIGCSE Technical Symposium on CS Education (St. Louis, Missouri, USA) (SIGCSE ’05). ACM, New York, NY, USA, 321–322. https://doi.org/10.1145/1047344.1047455
[13]
Irwin L Goldstein. 1993. Training in organizations: Needs assessment, development, and evaluation, 3rd ed.Thomson Brooks/Cole Publishing Co, Belmont, CA, US. xvi, 364–xvi, 364 pages.
[14]
Francis Heylighen, Iavor Kostov, and Mixel Kiemen. 2013. Mobilization Systems: technologies for motivating and coordinating human action. The New Development Paradigm: Education, Knowledge Economy and Digital Futures. Routledge. Retrieved from http://pcp. vub. ac. be/Papers/MobilizationSystems. pdf (2013).
[15]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2020. Gamification of student peer review in education: A systematic literature review. Education and Information Technologies(2020). https://doi.org/10.1007/s10639-020-10228-x
[16]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2020. A Review of Peer Code Review in Higher Education. ACM Trans. Comput. Educ. 20, 3, Article 22 (Sept. 2020), 25 pages. https://doi.org/10.1145/3403935
[17]
Michael S. Irwin and Stephen H. Edwards. 2019. Can Mobile Gaming Psychology Be Used to Improve Time Management on Programming Assignments?. In Proceedings of the ACM Conference on Global Computing Education (Chengdu,Sichuan, China) (CompEd ’19). ACM, New York, NY, USA, 208–214. https://doi.org/10.1145/3300115.3309517
[18]
Shivam Khandelwal, Sai Krishna Sripada, and Y. Raghu Reddy. 2017. Impact of Gamification on Code Review Process: An Experimental Study. In Proceedings of the 10th Innovations in Software Engineering Conference (Jaipur, India) (ISEC ’17). ACM, New York, NY, USA, 122–126. https://doi.org/10.1145/3021460.3021474
[19]
Z. Kubincová and M. Homola. 2017. Code review in computer science courses: Take one. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 10473 LNCS (2017), 125–135. https://doi.org/10.1007/978-3-319-66733-1_14 cited By 1.
[20]
James A. Kulik and Chen-Lin C. Kulik. 1988. Timing of Feedback and Verbal Learning. Review of Educational Research 58, 1 (1988), 79–97. https://doi.org/10.3102/00346543058001079 arXiv:https://doi.org/10.3102/00346543058001079
[21]
Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15). ACM, New York, NY, USA, 75–84. https://doi.org/10.1145/2724660.2724670
[22]
Elias Kyewski and Nicole C. Krämer. 2018. To gamify or not to gamify? An experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers & Education 118 (2018), 25 – 37. https://doi.org/10.1016/j.compedu.2017.11.006
[23]
Nikoletta-Zampeta Legaki, Nannan Xi, Juho Hamari, Kostas Karpouzis, and Vassilios Assimakopoulos. 2020. The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer Studies 144 (2020), 102496. https://doi.org/10.1016/j.ijhcs.2020.102496
[24]
X. Li. 2006. Using Peer Review to Assess Coding Standards - A Case Study. In Proceedings. Frontiers in Education. 36th Annual Conference. 9–14. https://doi.org/10.1109/FIE.2006.322572
[25]
Andrew Luxton-Reilly, Arthur Lewis, and Beryl Plimmer. 2018. Comparing Sequential and Parallel Code Review Techniques for Formative Feedback. In Proceedings of the 20th Australasian Computing Education Conference (Brisbane, Queensland, Australia) (ACE ’18). ACM, New York, NY, USA, 45–52. https://doi.org/10.1145/3160489.3160498
[26]
Joshua Martin, Stephen H. Edwards, and Clfford A. Shaffer. 2015. The Effects of Procrastination Interventions on Programming Project Success. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (Omaha, Nebraska, USA) (ICER ’15). ACM, New York, NY, USA, 3–11. https://doi.org/10.1145/2787622.2787730
[27]
Elisa D. Mekler, Florian Brühlmann, Klaus Opwis, and Alexandre N. Tuch. 2013. Do Points, Levels and Leaderboards Harm Intrinsic Motivation? An Empirical Analysis of Common Gamification Elements. In Proceedings of the First International Conference on Gameful Design, Research, and Applications (Toronto, Ontario, Canada) (Gamification ’13). ACM, New York, NY, USA, 66–73. https://doi.org/10.1145/2583008.2583017
[28]
Gabriela Morales-Martinez, Paul Latreille, and Paul Denny. 2020. Nationality and Gender Biases in Multicultural Online Learning Environments: The Effects of Anonymity. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376283
[29]
Manuel Palomo-Duarte, Juan Manuel Dodero, and Antonio García-Domínguez. 2014. Betting System for Formative Code Review in Educational Competitions. Expert Syst. Appl. 41, 5 (April 2014), 2222–2230. https://doi.org/10.1016/j.eswa.2013.09.020
[30]
Joe Gibbs Politz, Joseph M. Collard, Arjun Guha, Kathi Fisler, and Shriram Krishnamurthi. 2016. The Sweep: Essential Examples for In-Flow Peer Review. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (Memphis, Tennessee, USA) (SIGCSE ’16). ACM, New York, NY, USA, 243–248. https://doi.org/10.1145/2839509.2844626
[31]
Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). ACM, New York, NY, USA, 202–212. https://doi.org/10.1145/2491411.2491444
[32]
R.H. Rizzardini, M.M. Chan, and C. Guetl. 2016. Chapter 14 - An Attrition Model for MOOCs: Evaluating the Learning Strategies of Gamification. In Formative Assessment, Learning Data Analytics and Gamification, Santi Caballé and Robert Clarisó (Eds.). Academic Press, Boston, 295 – 311. https://doi.org/10.1016/B978-0-12-803637-2.00014-2
[33]
Arlene A Russell. 2004. Calibrated peer review-a writing and critical-thinking instructional tool. Teaching Tips: Innovations in Undergraduate Science Instruction 54 (2004).
[34]
Harald Søndergaard and Raoul A. Mulder. 2012. Collaborative learning through formative peer review: pedagogy, programs and potential. Computer Science Education 22, 4 (2012), 343–367. https://doi.org/10.1080/08993408.2012.728041 arXiv:https://doi.org/10.1080/08993408.2012.728041
[35]
Xiangyu Song, Seth Copen Goldstein, and Majd Sakr. 2020. Using Peer Code Review as an Educational Tool. In Proc. of the 2020 ACM Conf. on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). ACM, New York, NY, USA, 173–179. https://doi.org/10.1145/3341525.3387370
[36]
T. Stalhane, C. Kutay, H. Al-Kilidar, and R. Jeffery. 2004. Teaching the process of code review. In 2004 Australian Software Engineering Conference. Proceedings.271–278. https://doi.org/10.1109/ASWEC.2004.1290480
[37]
Martijn Stegeman, Erik Barendsen, and Sjaak Smetsers. 2014. Towards an empirically validated model for assessment of code quality. In Proc. the 14th Koli Calling international conference on computing education research. 99–108.
[38]
Keith Topping. 1998. Peer Assessment Between Students in Colleges and Universities. Review of Educational Research 68, 3 (1998), 249–276. https://doi.org/10.3102/00346543068003249 arXiv:https://doi.org/10.3102/00346543068003249
[39]
Crystal Han-Huei Tsay, Alexander K. Kofinas, Smita K. Trivedi, and Yang Yang. 2020. Overcoming the novelty effect in online gamified learning systems: An empirical evaluation of student engagement and performance. Journal of Computer Assisted Learning 36, 2 (2020), 128–146. https://doi.org/10.1111/jcal.12385
[40]
Y. Wang, H. Li, Y. Feng, Y. Jiang, and Y. Liu. 2012. Assessment of programming language learning based on peer code review model: Implementation and experience report. Computers & Education 59, 2 (2012), 412–422. https://doi.org/10.1016/j.compedu.2012.01.007 cited By 43.
[41]
Yanqing WANG, LI Yijun, Michael Collins, and Peijie LIU. 2008. Process Improvement of Peer Code Review and Behavior Analysis of Its Participants. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’08). ACM, New York, NY, USA, 107–111. https://doi.org/10.1145/1352135.1352171
[42]
W. Yanqing, L. Hang, S. Yanan, J. Yu, and Y. Jie. 2011. Learning outcomes of programming language courses based on peer code review model. In 2011 6th International Conference on Computer Science Education (ICCSE). 751–754. https://doi.org/10.1109/ICCSE.2011.6028746
[43]
Eric Zimmerling, Christoph E. Höllig, Philipp G. Sandner, and Isabell M. Welpe. 2019. Exploring the influence of common game elements on ideation output and motivation. Journal of Business Research 94 (2019), 302 – 312. https://doi.org/10.1016/j.jbusres.2018.02.030

Cited By

View all
  • (2025)An International Examination of Non-Technical Skills and Professional Dispositions in Computing -- Identifying the Present Day Academia-Industry Gap2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709610(124-174)Online publication date: 22-Jan-2025
  • (2024)Applying a New Game Element Called Backward Grading For Student EngagementApplying a new game element called backward grading for student engagementRevista Educación en Ingeniería10.26507/rei.v19n37.127819:37(1-8)Online publication date: 16-Jan-2024
  • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
  • Show More Cited By

Index Terms

  1. Improving Student Peer Code Review Using Gamification
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        ACE '21: Proceedings of the 23rd Australasian Computing Education Conference
        February 2021
        195 pages
        ISBN:9781450389761
        DOI:10.1145/3441636
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 17 March 2021

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. CS1
        2. code review
        3. computing education
        4. gamification
        5. higher education
        6. peer code review
        7. peer review
        8. programming course

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        • Indonesian Endowment Fund for Education (LPDP)

        Conference

        ACE '21
        ACE '21: Australasian Computing Education Conference
        February 2 - 4, 2021
        SA, Virtual, Australia

        Acceptance Rates

        Overall Acceptance Rate 161 of 359 submissions, 45%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)59
        • Downloads (Last 6 weeks)2
        Reflects downloads up to 23 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)An International Examination of Non-Technical Skills and Professional Dispositions in Computing -- Identifying the Present Day Academia-Industry Gap2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709610(124-174)Online publication date: 22-Jan-2025
        • (2024)Applying a New Game Element Called Backward Grading For Student EngagementApplying a new game element called backward grading for student engagementRevista Educación en Ingeniería10.26507/rei.v19n37.127819:37(1-8)Online publication date: 16-Jan-2024
        • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
        • (2024)Towards an effective methodology for the integral gamification of classesMultimedia Tools and Applications10.1007/s11042-024-19846-wOnline publication date: 18-Jul-2024
        • (2023)Impacting the Submission Timing of Student Work Using GamificationProceedings of the 16th Annual ACM India Compute Conference10.1145/3627217.3627218(7-12)Online publication date: 9-Dec-2023
        • (2023)Learning from Errors: An Empirical Study on the Impact of Gamification on SQL Query FormulationProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588821(341-347)Online publication date: 29-Jun-2023
        • (2022)Implementation of Gamification in Programming Learning: Literature ReviewInternational Journal of Information and Education Technology10.18178/ijiet.2022.12.12.177112:12(1448-1457)Online publication date: 2022
        • (2022)Can Students Review Their Peers?Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 110.1145/3502718.3524762(12-18)Online publication date: 7-Jul-2022
        • (2022)A Comparison of Immediate and Scheduled Feedback in Introductory Programming ProjectsProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 110.1145/3478431.3499372(885-891)Online publication date: 22-Feb-2022
        • (2022)Developing a system to increase motivation and engagement in student code peer review2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)10.1109/TALE54877.2022.00023(93-98)Online publication date: Dec-2022
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media