skip to main content
10.1145/2493394.2493410acmconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article

In-game assessments increase novice programmers' engagement and level completion speed

Published: 12 August 2013 Publication History

Abstract

Assessments have been shown to have positive effects on learning in compulsory educational settings. However, much less is known about their effects in discretionary learning settings, especially in computing education and educational games. We hypothesized that adding assessments to an educational computing game would provide extra opportunities for players to practice and correct misconceptions, thereby affecting their performance on subsequent levels and their motivation to continue playing. To test this, we designed a game called Gidget, in which players help a robot find and fix defects in programs that follow a mastery learning paradigm. Across two studies, we manipulated the inclusion of multiple choice and self-explanation assessment levels in the game, measuring their impact on engagement and level completion speed. In our first study, we found that including assessments caused learners to voluntarily play longer and complete more levels, suggesting increased engagement; in our second study, we found that including assessments caused learners to complete levels faster, suggesting increased understanding. These findings suggest that including assessments in a discretionary computing education game may be a key design strategy for improving informal learning of computing concepts.

References

[1]
Aleven, V., & Koedinger, K.R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. CogSci, 26(2), 147--179.
[2]
Aleven, V., Myers, E., Easterday, M., & Ogan, A. (2010). Toward a framework for the analysis and design of educational games. IEEE DIGITEL, 69--76.
[3]
Bjork, R.A. (1999). Assessing our own competence: heuristics and illusions. In Gopher, D., & Koriat, A., Attention and performance XVII: Cognitive regulation of performance, 435--459. Cambridge, MA: MIT Press.
[4]
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in education, 5(1), 7--74.
[5]
Boustedt, J., Eckerdal, A., McCartney, R., Sanders, K., Thomas, L., & Zander C. (2011). Students' perceptions of the differences between formal and informal learning. ACM ICER, 61--68.
[6]
Butler, A.C., & Roediger, H.L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36(3), 604--616.
[7]
Campbell, J., & Mayer, R.E. (2009). Questioning as an instructional method: Does it affect learning from lectures. Applied Cognitive Psychology, 23, 747--759.
[8]
Carpenter, S.K., Pashler, H., & Vul, E. (2006). What types of learning are enhanced by a cued recall test? Psychonomic Bulletin & Review, 13, 826--830.
[9]
Chi, M.T., De Leeuw, N., Chiu, M.H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439--477.
[10]
Cross, J. (2006). Informal learning: rediscovering the natural pathways that inspire innovation and performance. San Francisco, CA: Pfeiffer.
[11]
Csikszentmihalyi, M. (1990). Flow: The psychology of optical experience. New York, NY: Harper Perrennial.
[12]
Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. JIME, 3.
[13]
Eagle, M., & Barnes, T. (2009). Experimental evaluation of an educational game for improved learning in introductory computing. ACM SIGCSE Bulletin, 41(1), 321--325.
[14]
Garris, R., Ahlers, R., & Driskell, J.E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 4, 441--467.
[15]
Gee, J.P. (2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan.
[16]
Hays, R.T. (2005). The effectiveness of instructional games: A literature review and discussion (Technical Report 2005-004). Naval air warfare ctr. training systems division. Orlando, FL.
[17]
Johnson, C.I., & Mayer, R.E. (2009). A testing effect with multimedia learning. J. Educ. Psychol., 101(3), 621--629.
[18]
Kapp, K.M. (2012). The gamification of learning and instruction: game-based methods and strategies for training and education. San Francisco, CA: Pfeiffer.
[19]
Karpicke, J.D., & Roediger, H.L. (2007). Expanding retrieval promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 704--719.
[20]
Karpicke, J.D., & Roediger, H.L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57, 151--162.
[21]
Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9), retrieved April 2013 from http://pareonline.net/getvn.asp?v=4&n=9
[22]
Lee, M.J., & Ko, A.J. (2011). Personifying programming tool feedback improves novice programmers' learning. ACM ICER, 109--116.
[23]
Lee, M.J., & Ko, A.J. (2012). Investigating the role of purposeful goals on novices' engagement in a programming game. IEEE VL/HCC, 163--166.
[24]
Malone, T.W. (1981). What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games. Palo Alto, CA: Xerox.
[25]
McDaniel, M.A., Anderson, J.L., Derbish, M.H., & Morrisette, N. (2007). Testing the testing effect in the classroom. European Journal of Cognitive Psychology, 19(4--5), 494--513.
[26]
McNamara, D., Jackson, G., Graesser, A. (2009) Intelligent tutoring and games. Artificial Intelligence in Education, 1--10.
[27]
Mislevy, R.J., Behrens, J.T., Dicerbo, K.E., Frezzo, D.C., & West, P. (2012). Three things game designers need to know about assessment. Assessment in game-based learning, 59--8
[28]
Pear, J.J. (2004). Enhanced feedback using computer-aided personalized system of instruction. In W. Buskist, V. W. Hevern, B.K. Saville, & T. Zinn, (Eds.), Essays from excellence in teaching (Chapter 11).
[29]
Poehner, M. E. (2007). Beyond the test: L2 dynamic assessment and the transcendence of mediated learning. The Modern Language Journal, 91, 323--340.
[30]
Randel, J.M., Morris, B.A., Wetzel, C.D., & Whitehill, B.V. (1992). The effectiveness of games for educational purposes: A review of recent research. Simulation & Gaming, 23(3), 261--276.
[31]
Riggio, R. E. (2007). Reciprocal peer tutoring: Learning through dyadic teaching. In B. K. Saville, T. E. Zinn, S. A. Meyers, & J. R. Stowell (Eds.), Essays from excellence in teaching, (Chapter 10).
[32]
Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., Tomlinson, B. (2010). Who are the crowdworkers?: Shifting demographics in Amazon Mechanical Turk. ACM CHI, 2863--2872.
[33]
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119--144.
[34]
Shute, V.J. (2011). Stealth assessment in computer-based games to support learning. Computer games and instruction, 55(2), 503--524.
[35]
Shute, V.J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. Serious games: Mechanisms and Effects, 295--321.
[36]
Smith, T. (2007). Exams as learning experiences: One nutty idea after another. Beyond Tests and Quizzes: Creative Assessments in the College Classroom, 115, 71.
[37]
Sweller, J. (2006). The worked example effect and human cognition. Learning and Instruction, 16(2) 165--169.
[38]
Vrugt, A.J., Langereis, M.P., & Hoogstraten, J. (1997). Academic self-efficacy and malleability of relevant capabilities as predictors of exam performance. Journal of Experimental Education, 66(1), 61--72.
[39]
Young, J. (2008). "Badges" earned online pose challenge to traditional college diplomas. Chronicle of Higher Education.

Cited By

View all
  • (2024)An In-Depth Evaluation of Educational Burst Games in Relation to Learner ProficiencyMultimodal Technologies and Interaction10.3390/mti81000888:10(88)Online publication date: 11-Oct-2024
  • (2024)Digital game-based learning: Pedagogical agent and feedback types on achievement, flow experience, and cognitive loadEducation and Information Technologies10.1007/s10639-023-12368-229:10(12943-12968)Online publication date: 1-Jul-2024
  • (2024)Coarse-Grained Detection for Personalized Online Learning InterventionsA Human-Centered Perspective of Intelligent Personalized Environments and Systems10.1007/978-3-031-55109-3_8(207-220)Online publication date: 1-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICER '13: Proceedings of the ninth annual international ACM conference on International computing education research
August 2013
202 pages
ISBN:9781450322430
DOI:10.1145/2493394
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2013

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. assessment
  2. debugging
  3. educational game
  4. engagement
  5. programming
  6. serious game
  7. speed

Qualifiers

  • Research-article

Conference

ICER '13
Sponsor:
ICER '13: International Computing Education Research Conference
August 12 - 14, 2013
San California, San Diego, USA

Acceptance Rates

ICER '13 Paper Acceptance Rate 22 of 70 submissions, 31%;
Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025
ACM Conference on International Computing Education Research
August 3 - 6, 2025
Charlottesville , VA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)36
  • Downloads (Last 6 weeks)9
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)An In-Depth Evaluation of Educational Burst Games in Relation to Learner ProficiencyMultimodal Technologies and Interaction10.3390/mti81000888:10(88)Online publication date: 11-Oct-2024
  • (2024)Digital game-based learning: Pedagogical agent and feedback types on achievement, flow experience, and cognitive loadEducation and Information Technologies10.1007/s10639-023-12368-229:10(12943-12968)Online publication date: 1-Jul-2024
  • (2024)Coarse-Grained Detection for Personalized Online Learning InterventionsA Human-Centered Perspective of Intelligent Personalized Environments and Systems10.1007/978-3-031-55109-3_8(207-220)Online publication date: 1-May-2024
  • (2022)Integrating Assessment, Feedback, and Learning Analytics in Educational GamesResearch Anthology on Developments in Gamification and Game-Based Learning10.4018/978-1-6684-3710-0.ch088(1803-1846)Online publication date: 2022
  • (2022)An Analysis of Middle Grade Teachers' Debugging Pedagogical Content KnowledgeProceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 110.1145/3502718.3524770(533-539)Online publication date: 7-Jul-2022
  • (2022)Learn ARP Spoofing Attack in a Game2022 IEEE Frontiers in Education Conference (FIE)10.1109/FIE56618.2022.9962649(1-5)Online publication date: 8-Oct-2022
  • (2022) ML-Quest : a game for introducing machine learning concepts to K-12 students Interactive Learning Environments10.1080/10494820.2022.208411532:1(229-244)Online publication date: 8-Jun-2022
  • (2021)Autonomy-supportive game benefits both inexperienced and experienced programmersJournal of Computing Sciences in Colleges10.5555/3503984.350399337:2(89-97)Online publication date: 3-Dec-2021
  • (2021)Auto-generated game levels increase novice programmers' engagementJournal of Computing Sciences in Colleges10.5555/3447080.344708936:3(70-79)Online publication date: 27-Jan-2021
  • (2021)To Be or Not to Be Stuck, or Is It a Continuum?Proceedings of the ACM on Human-Computer Interaction10.1145/34746565:CHI PLAY(1-35)Online publication date: 6-Oct-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media