Benjamin Xie
Jared Ordona Lim
Amy J. Ko
ABSTRACT
Learning programming benefits from self-regulation, but novices lack support for developing these skills of cognitive control. To support their development, we designed Code Replayer, an online tool that enables novice programmers to practice programming and then replay their coding process to reflect and identify process improvements. To evaluate the impact of replaying code on self-regulation, we conducted a formative qualitative evaluation with 21 novice programmers who used Code Replayer to practice writing code. We found that after watching code replays, participants more frequently interpreted problem prompts and planned their solutions, two crucial self-regulation behaviors that novices often overlook. We interpret our results by focusing on two focal points in the design of code replays as a programming self-regulation intervention: interpreting pauses in replays and ensuring replays of struggle are more informative and less detrimental.
Supplemental Material
- Serhat Altıok, Zeynep Başer, and Erman Yükseltürk. 2019. Enhancing metacognitive awareness of undergraduates through using an e-educational video environment. Comput. Educ. 139 (Oct. 2019), 129–145. https://doi.org/10.1016/j.compedu.2019.05.010Google Scholar
Digital Library
- Mark Aveline. 1992. The use of audio and videotape recordings of therapy sessions in the supervision and practice of dynamic psychotherapy. British journal of psychotherapy 8, 4 (June 1992), 347–358. https://doi.org/10.1111/j.1752-0118.1992.tb01198.xGoogle Scholar
- Ryan Baker and Aaron Hawn. 2021. Algorithmic Bias In Education. (2021). https://doi.org/10.35542/osf.io/pbmvzGoogle Scholar
- Alan F Blackwell. 2002. What is Programming?. In Psychology of Programming Interest Group (PPIG).Google Scholar
- Jonas Boustedt, Anna Eckerdal, Robert McCartney, Kate Sanders, Lynda Thomas, and Carol Zander. 2011. Students’ perceptions of the differences between formal and informal learning. In Proceedings of the seventh international workshop on Computing education research. ACM. https://doi.org/10.1145/2016911.2016926Google ScholarDigital Library
- Glenn A Bowen. 2006. Grounded Theory and Sensitizing Concepts. International Journal of Qualitative Methods 5, 3 (Sept. 2006), 12–23. https://doi.org/10.1177/160940690600500304Google ScholarCross Ref
- Michelene T H Chi. 1997. Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. Journal of the Learning Sciences 6, 3 (July 1997), 271–315. https://doi.org/10.1207/s15327809jls0603_1Google ScholarCross Ref
- Nicola Dell, Vidya Vaidyanathan, Indrani Medhi, Edward Cutrell, and William Thies. 2012. Yours is better!. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin Texas USA). ACM, New York, NY, USA. https://doi.org/10.1145/2207676.2208589Google ScholarDigital Library
- Joseph Ditton, Hillary Swanson, and John Edwards. 2021. External Imagery in Computer Programming. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 1226–1231. https://doi.org/10.1145/3408877.3432426Google ScholarDigital Library
- Karl Anders Ericsson and Herbert Alexander Simon. 1993. Protocol Analysis: Verbal Reports as Data Revised Edition. The MIT Press.Google ScholarCross Ref
- Anneli Eteläpelto. 1993. Metacognition and the Expertise of Computer Program Comprehension. Scandinavian Journal of Educational Research 37, 3 (Jan. 1993), 243–254. https://doi.org/10.1080/0031383930370305Google ScholarCross Ref
- Gordon Fjeldsted and John Edwards. 2022. Quantifying Student Struggles using Heatmaps and Keystroke Data. In 2022 Intermountain Engineering, Technology and Computing (IETC). 1–5. https://doi.org/10.1109/IETC54973.2022.9796894Google Scholar
- Enrico Gandolfi, Richard E Ferdig, and Robert Clements. 2022. Streaming code across audiences and performers: An analysis of computer science communities of inquiry on Twitch.tv. British journal of educational technology: journal of the Council for Educational Technology (Feb. 2022). https://doi.org/10.1111/bjet.13207Google Scholar
- Nichole M Garcia, Nancy López, and Verónica N Vélez. 2018. QuantCrit: rectifying quantitative methods through critical race theory. Race Ethnicity and Education 21, 2 (March 2018), 149–157. https://doi.org/10.1080/13613324.2017.1377675Google ScholarCross Ref
- Philip J Guo. 2013. Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 579–584. https://doi.org/10.1145/2445196.2445368Google ScholarDigital Library
- Lassi Haaranen. 2017. Programming as a Performance: Live-streaming and Its Implications for Computer Science Education. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (Bologna, Italy) (ITiCSE ’17). Association for Computing Machinery, New York, NY, USA, 353–358. https://doi.org/10.1145/3059009.3059035Google ScholarDigital Library
- David Hammer and Leema K Berland. 2014. Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning. Journal of the Learning Sciences 23, 1 (Jan. 2014), 37–46. https://doi.org/10.1080/10508406.2013.802652Google ScholarCross Ref
- Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. Most people are not WEIRD. Nature 466, 7302 (July 2010), 29. https://doi.org/10.1038/466029aGoogle ScholarCross Ref
- William T Hoyt. 2010. Interrater reliability and agreement. In The Reviewer’s Guide to Quantitative Methods in the Social Sciences, Gregory R Hancock, Ralph O Mueller, and Laura M Stapleton (Eds.). Routledge, 141–154. https://doi.org/10.4324/9781315755649-10/interrater-reliability-agreement-william-hoytGoogle Scholar
- C M Janelle, D A Barba, S G Frehlich, L K Tennant, and J H Cauraugh. 1997. Maximizing performance feedback effectiveness through videotape replay and a self-controlled learning environment. Research quarterly for exercise and sport 68, 4 (Dec. 1997), 269–279. https://doi.org/10.1080/02701367.1997.10608008Google Scholar
- Slava Kalyuga. 2009. The Expertise Reversal Effect. In Managing Cognitive Load in Adaptive Multimedia Learning. IGI Global, 58–80. https://doi.org/10.4018/978-1-60566-048-6.ch003Google Scholar
- Harrison Kwik, Benjamin Xie, and Amy J Ko. 2018. Experiences of Computer Science Transfer Students. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). ACM Press, 115–123. https://doi.org/10.1145/3230977.3231004Google ScholarDigital Library
- Juho Leinonen, Leo Leppänen, Petri Ihantola, and Arto Hellas. 2017. Comparison of Time Metrics in Programming. In Proceedings of the 2017 ACM Conference on International Computing Education Research(ICER ’17). ACM, New York, NY, USA, 200–208. https://doi.org/10.1145/3105726.3106181Google ScholarDigital Library
- Leo Leppänen, Juho Leinonen, and Arto Hellas. 2016. Pauses and spacing in learning to program. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’16). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/2999541.2999549Google ScholarDigital Library
- Colleen M Lewis. 2012. The importance of students’ attention to program state: a case study of debugging behavior. In Proceedings of the ninth annual international conference on International computing education research (Auckland, New Zealand) (ICER ’12). Association for Computing Machinery, New York, NY, USA, 127–134. https://doi.org/10.1145/2361276.2361301Google ScholarDigital Library
- Colleen M Lewis and Niral Shah. 2015. How Equity and Inequity Can Emerge in Pair Programming. In Proceedings of the eleventh annual International Conference on International Computing Education Research - ICER ’15. ACM Press, Omaha, Nebraska, USA, 41–50. https://doi.org/10.1145/2787622.2787716Google ScholarDigital Library
- Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI?. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21, Article 143). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3411764.3445488Google ScholarDigital Library
- Rebecca Lippmann Kung and Cedric Linder. 2007. Metacognitive activity in the physics student laboratory: is increased metacognition necessarily better?Metacognition and Learning 2, 1 (April 2007), 41–56. https://doi.org/10.1007/s11409-007-9006-9Google Scholar
- Ye Liu and Xiaolan Fu. 2007. How Does Distraction Task Influence the Interaction of Working Memory and Long-Term Memory? In Engineering Psychology and Cognitive Ergonomics, Don Harris (Ed.). Vol. 4562. Springer Berlin Heidelberg, Berlin, Heidelberg, 366–374.Google Scholar
- Samuel A Livingston and Michael J Zieky. 1982. Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Educational Testing Service.Google Scholar
- Samuel A Livingston and Michael J Zieky. 1989. A Comparative Study of Standard-Setting Methods. Applied Measurement in Education 2, 2 (April 1989), 121–141. https://doi.org/10.1207/s15324818ame0202_3Google ScholarCross Ref
- Dastyni Loksa and Amy J Ko. 2016. The Role of Self-Regulation in Programming Problem Solving Process and Success. In Proceedings of the 2016 ACM Conference on International Computing Education Research(ICER ’16). ACM, New York, NY, USA, 83–91. https://doi.org/10.1145/2960310.2960334Google ScholarDigital Library
- Dastyni Loksa, Amy J Ko, Will Jernigan, Alannah Oleson, Christopher J Mendez, and Margaret M Burnett. 2016. Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance. ACM Press, 1449–1461. https://doi.org/10.1145/2858036.2858252Google ScholarDigital Library
- Dastyni Loksa, Lauren Margulieux, Brett A Becker, Michelle Craig, Paul Denny, Raymond Pettit, and James Prather. 2021. Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use. ACM Trans. Comput. Educ. (Dec. 2021). https://doi.org/10.1145/3487050Google ScholarDigital Library
- Dastyni Loksa, Benjamin Xie, Harrison Kwik, and Amy J Ko. 2020. Investigating Novices’ In Situ Reflections on Their Programming Process. In Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE), Research Track. ACM.Google ScholarDigital Library
- Murali Mani and Quamrul Mazumder. 2013. Incorporating metacognition into learning. In Proceeding of the 44th ACM technical symposium on Computer science education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 53–58. https://doi.org/10.1145/2445196.2445218Google ScholarDigital Library
- Robert Lindsay McWilliams. 1996. An Investigation Into the Use and Effectiveness of Videotape Self-evaluations of Conducting for Practicing Music Educators. University of Minnesota.Google Scholar
- Laurie Murphy, Sue Fitzgerald, Brian Hanks, and Renée McCauley. 2010. Pair debugging: a transactive discourse analysis. In Proceedings of the Sixth international workshop on Computing education research (Aarhus, Denmark) (ICER ’10). Association for Computing Machinery, New York, NY, USA, 51–58. https://doi.org/10.1145/1839594.1839604Google ScholarDigital Library
- Alannah Oleson, Benjamin Xie, Jean Salac, Jayne Everson, F Megumi Kivuva, and Amy J Ko. 2022. A Decade of Demographics in Computing Education Research: A Critical Review of Trends in Collection, Reporting, and Use. In Proceedings of the 2022 ACM Conference on International Computing Education Research (Lugano and Virtual Event, Switzerland) (ICER 2022). ACM. https://doi.org/10.1145/3501385.3543967Google ScholarDigital Library
- Ernesto Panadero, Julia Klug, and Sanna Järvelä. 2016. Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research 60, 6 (Nov. 2016), 723–735. https://doi.org/10.1080/00313831.2015.1066436Google ScholarCross Ref
- Luc Paquette, Jaclyn Ocumpaugh, Ziyue Li, Alexandra Andres, and Ryan Baker. 2020. Who’s Learning? Using Demographics in EDM Research. Journal of Educational Data Mining 12, 3 (2020), 1–30.Google Scholar
- Nick Parlante. 2017. Python Practice. https://codingbat.com/python. Accessed: 2022-4-20.Google Scholar
- Michael Quinn Patton. 2014. Qualitative Research & Evaluation Methods: Integrating Theory and Practice. SAGE Publications.Google Scholar
- John Paxton. 2002. Live programming as a lecture technique. J. Comput. Sci. Coll. 18, 2 (Dec. 2002), 51–56.Google ScholarDigital Library
- Paul R Pintrich and And Others. 1991. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ).Google Scholar
- Paul R Pintrich and Elisabeth V de Groot. 1990. Motivational and self-regulated learning components of classroom academic performance. Journal of educational psychology 82, 1 (March 1990), 33–40. https://doi.org/10.1037/0022-0663.82.1.33Google ScholarCross Ref
- Paul R Pintrich, David A F Smith, Teresa Garcia, and Wilbert J Mckeachie. 1993. Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq). Educational and psychological measurement 53, 3 (Sept. 1993), 801–813. https://doi.org/10.1177/0013164493053003024Google Scholar
- James Prather, Lauren Margulieux, Jacqueline Whalley, Paul Denny, Brent N Reeves, Brett A Becker, Paramvir Singh, Garrett Powell, and Nigel Bosch. 2022. Getting By With Help From My Friends: Group Study in Introductory Programming Understood as Socially Shared Regulation. https://doi.org/10.1145/3501385.3543970Google ScholarDigital Library
- James Prather, Raymond Pettit, Brett A Becker, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht, and Krista Masci. 2019. First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 531–537. https://doi.org/10.1145/3287324.3287374Google ScholarDigital Library
- Thomas W Price, David Hovemeyer, Kelly Rivers, Ge Gao, Austin Cory Bart, Ayaan M Kazerouni, Brett A Becker, Andrew Petersen, Luke Gusukuma, Stephen H Edwards, and David Babcock. 2020. ProgSnap2: A Flexible Format for Programming Process Data. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 356–362. https://doi.org/10.1145/3341525.3387373Google ScholarDigital Library
- Alex Radermacher, Gursimran Walia, and Richard Rummelt. 2012. Improving student learning outcomes with pair programming. In Proceedings of the ninth annual international conference on International computing education research (Auckland, New Zealand) (ICER ’12). Association for Computing Machinery, New York, NY, USA, 87–92. https://doi.org/10.1145/2361276.2361294Google ScholarDigital Library
- Adalbert Gerald Soosai Raj, Jignesh M Patel, Richard Halverson, and Erica Rosenfeld Halverson. 2018. Role of Live-coding in Learning Introductory Programming. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’18, Article 13). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3279720.3279725Google ScholarDigital Library
- Fernando J Rodríguez, Kimberly Michelle Price, and Kristy Elizabeth Boyer. 2017. Exploring the Pair Programming Process: Characteristics of Effective Collaboration. ACM Press, 507–512. https://doi.org/10.1145/3017680.3017748Google ScholarDigital Library
- Barbara Rogoff, Maureen Callanan, Kris D Gutiérrez, and Frederick Erickson. 2016. The Organization of Informal Learning. Review of Research in Education 40, 1 (March 2016), 356–401. https://doi.org/10.3102/0091732X16680994Google ScholarCross Ref
- Marc J Rubin. 2013. The effectiveness of live-coding to teach introductory programming. In Proceeding of the 44th ACM technical symposium on Computer science education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 651–656. https://doi.org/10.1145/2445196.2445388Google ScholarDigital Library
- Bernhard Schmitz and Franziska Perels. 2011. Self-monitoring of self-regulation during math homework behaviour using standardized diaries., 255–273 pages. https://doi.org/10.1007/s11409-011-9076-6Google Scholar
- Alan H Schoenfeld. 1992. On paradigms and methods: What do you do when the ones you know don’t do what you want them to? Issues in the analysis of data in the form of videotapes. Journal of the Learning Sciences 2, 2 (April 1992), 179–214. https://doi.org/10.1207/s15327809jls0202_3Google ScholarCross Ref
- Ana Selvaraj, Eda Zhang, Leo Porter, and Adalbert Gerald Soosai Raj. 2021. Live Coding: A Review of the Literature. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 (Virtual Event, Germany) (ITiCSE ’21). Association for Computing Machinery, New York, NY, USA, 164–170. https://doi.org/10.1145/3430665.3456382Google ScholarDigital Library
- Raj Shrestha, Juho Leinonen, Arto Hellas, Petri Ihantola, and John Edwards. 2022. CodeProcess Charts: Visualizing the Process of Writing Code. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 46–55. https://doi.org/10.1145/3511861.3511867Google ScholarDigital Library
- Katta Spiel, Oliver Haimson, and Danielle Lottridge. 2019. How to do better with gender on surveys: A guide for HCI researchers. ACM Interactions 26, 4 (2019).Google ScholarDigital Library
- Michael A Stadler. 1995. Role of attention in implicit learning. Journal of experimental psychology. Learning, memory, and cognition 21, 3 (May 1995), 674–685. https://doi.org/10.1037/0278-7393.21.3.674Google ScholarCross Ref
- Claude Steele. 2011. Stereotype Threat and African-American Student Achievement. In The Inequality Reader (2 ed.). Routledge, 276–281. https://doi.org/10.4324/9780429494468-31Google Scholar
- S Stumpf, A Peters, S Bardzell, M Burnett, D Busse, J Cauchard, and E Churchill. 2020. Gender-Inclusive HCI Research and Design: A Conceptual Review. Foundations and Trends in Human–Computer Interaction 13, 1 (March 2020), 1–69. https://doi.org/10.1561/1100000056Google ScholarDigital Library
- S Christian Wheeler and Richard E Petty. 2001. The effects of stereotype activation on behavior: A review of possible mechanisms. Psychological bulletin 127, 6 (Nov. 2001), 797–826. https://doi.org/10.1037/0033-2909.127.6.797Google Scholar
- Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Amy J Ko. 2019. A theory of instruction for introductory programming skills. Computer Science Education (Jan. 2019), 1–49. https://doi.org/10.1080/08993408.2019.1565235Google Scholar
- Lisa Yan, Annie Hu, and Chris Piech. 2019. Pensieve: Feedback on Coding Process for Novices. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 253–259. https://doi.org/10.1145/3287324.3287483Google ScholarDigital Library
- Qian Zhang and Teomara Rutherford. 2022. Grade 5 students’ elective replay after experiencing failures in learning fractions in an educational game: When does replay after failures benefit learning?. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online USA). ACM, New York, NY, USA. https://doi.org/10.1145/3506860.3506873Google ScholarDigital Library
- Barry J Zimmerman. 2000. Attaining Self-Regulation: A Social Cognitive Perspective. In Handbook of Self-Regulation, Monique Boekaerts, Paul R Pintrich, and Moshe Zeidner (Eds.). Academic Press, San Diego, 13–39. https://doi.org/10.1016/B978-012109890-2/50031-7Google ScholarCross Ref
- Barry J Zimmerman. 2002. Becoming a Self-Regulated Learner: An Overview. Theory into practice 41, 2 (May 2002), 64–70. https://doi.org/10.1207/s15430421tip4102_2Google Scholar
- Barry J Zimmerman and Anastasia Kitsantas. 1999. Acquiring writing revision skill: Shifting from process to outcome self-regulatory goals. Journal of educational psychology 91, 2 (1999), 241–250. https://doi.org/10.1037/0022-0663.91.2.241Google ScholarCross Ref
- Barry J Zimmerman and Adam R Moylan. 2009. Self-Regulation: Where Metacognition and Motivation Intersect. In Handbook of metacognition in education. Routledge, 311–328.Google Scholar
Index Terms
- Developing Novice Programmers’ Self-Regulation Skills with Code Replays
Recommendations
Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Metacognition and self-regulation are important skills for successful learning and have been discussed and researched extensively in the general education literature for several decades. More recently, there has been growing interest in understanding how ...
Are Pedagogical Agents' External Regulation Effective in Fostering Learning with Intelligent Tutoring Systems?
ITS 2016: Proceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 9684In this study we tested whether external regulation provided by artificial pedagogical agents PAs was effective in facilitating learners' self-regulated learning SRL and can therefore foster complex learning with a hypermedia-based intelligent tutoring ...
Visualizing Code Patterns in Novice Programmers
WCCCE '18: Proceedings of the 23rd Western Canadian Conference on Computing EducationMany researchers have investigated the difficulties faced by novice programmers. However, these approaches have so far focused primarily on the identification and correction of common syntax errors, or that of topic difficulty in the CS1 curriculum. ...
Comments