Anshul Shah
Leo Porter
William G. Griswold
ABSTRACT
Background and Context. Live coding is a teaching method in which an instructor dynamically writes code in front of students in an effort to impart skills such as incremental development and debugging. By contrast, traditional, static-code examples typically involve an instructor annotating or explaining components of pre-written code. Despite recommendations to use live coding and a wealth of qualitative analyses that identify perceived learning benefits of it, there are a lack of empirical evaluations to confirm those learning benefits, especially with respect to students’ programming processes.
Objectives. Our work aims to provide a holistic, empirical comparison of a live-coding pedagogy with a static-code one. We evaluated the impact of a live-coding pedagogy on three main areas: 1) students’ adherence to effective programming processes, 2) their performance on exams and assignments, and 3) their lecture experiences, such as engagement during lecture and perceptions of code examples.
Method. In our treatment-control quasi-experimental setup, one lecture group saw live-coding examples while the other saw only static-code ones. Both lecture groups were taught by the same instructor, were taught the exact same content, and completed the same assignments and exams. We collected compilation-level programming process data, student performance on exam and homework questions, and feedback via a survey and course evaluations.
Findings. Our findings showed no statistically significant differences between the live-coding and static-code groups on programming process metrics related to incremental development, debugging, and productivity. Similarly, there was no difference between the groups on course performance on assignments and exams. Finally, student feedback suggests that more students in the live-coding group reported that lectures were too fast and failed to facilitate note-taking, potentially mitigating the perceived benefits of live coding.
Implications. Live coding alone may not lead to many of the perceived and intended benefits that prior work identifies, but future work may investigate how to realize these benefits while minimizing the drawbacks we identified.
- M Aickin and H Gensler. 1996. Adjusting for multiple testing when reporting research results: The Bonferroni vs Holm methods.American Journal of Public Health 86, 5 (1996), 726–728. https://doi.org/10.2105/ajph.86.5.726Google Scholar
- Albert Bandura. 1977. Social Learning theory. Prentice-Hall, Englewood Cliffs, N.J.Google Scholar
- Brett A. Becker. 2016. A New Metric to Quantify Repeated Compiler Errors for Novice Programmers. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE ’16). Association for Computing Machinery, New York, NY, USA, 296–301. https://doi.org/10.1145/2899415.2899463Google ScholarDigital Library
- Jens Bennedsen and Michael E. Caspersen. 2005. Revealing the Programming Process. SIGCSE Bull. 37, 1 (feb 2005), 186–190. https://doi.org/10.1145/1047124.1047413Google ScholarDigital Library
- Naomi R. Boyer, Sara Langevin, and Alessio Gaspar. 2008. Self Direction & Constructivism in Programming Education. In Proceedings of the 9th ACM SIGITE Conference on Information Technology Education (Cincinnati, OH, USA) (SIGITE ’08). Association for Computing Machinery, New York, NY, USA, 89–94. https://doi.org/10.1145/1414558.1414585Google ScholarDigital Library
- Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International Journal of Man-Machine Studies 18, 6 (1983), 543–554. https://doi.org/10.1016/S0020-7373(83)80031-5Google ScholarCross Ref
- Neil C. C. Brown and Greg Wilson. 2018. Ten quick tips for teaching programming. PLOS Computational Biology 14, 4 (04 2018), 1–8. https://doi.org/10.1371/journal.pcbi.1006023Google Scholar
- Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2015. The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (Omaha, Nebraska, USA) (ICER ’15). Association for Computing Machinery, New York, NY, USA, 141–150. https://doi.org/10.1145/2787622.2787710Google ScholarDigital Library
- Charis Charitsis, Chris Piech, and John C. Mitchell. 2022. Using NLP to Quantify Program Decomposition in CS1. In Proceedings of the Ninth ACM Conference on Learning @ Scale (New York City, NY, USA) (L@S ’22). Association for Computing Machinery, New York, NY, USA, 113–120. https://doi.org/10.1145/3491140.3528272Google ScholarDigital Library
- Jacob Cohen. 1977. Statistical Power Analysis for the behavioral sciences. Academic Press.Google Scholar
- Allan Collins, John Seely Brown, Ann Holum, 1991. Cognitive apprenticeship: Making thinking visible. American educator 15, 3 (1991), 6–11.Google Scholar
- Allan M. Collins, John Seely Brown, and Susan E. Newman. 1988. Cognitive Apprenticeship: Teaching the Crafts of Reading, Writing, and Mathematics. Knowing, Learning, and Instruction 8, 1 (1988), 2–10. https://doi.org/10.5840/thinking19888129Google Scholar
- Edward E. Cureton. 1956. Rank-biserial correlation. Psychometrika 21 (1956), 287–290.Google ScholarCross Ref
- Leon R. de Bruin. 2019. The use of cognitive apprenticeship in the learning and teaching of improvisation: Teacher and student perspectives. Research Studies in Music Education 41, 3 (2019), 261–279. https://doi.org/10.1177/1321103X18773110Google ScholarCross Ref
- Vincenzo Del Fatto, Gabriella Dodero, and Rosella Gennari. 2016. How Measuring Student Performances Allows for Measuring Blended Extreme Apprenticeship for Learning Bash Programming. Comput. Hum. Behav. 55, PB (feb 2016), 1231–1240.Google Scholar
- Sarah LS Duncan. 1996. Cognitive apprenticeship in classroom instruction: Implications for industrial and technical teacher education. Journal of Industrial Teacher Education 33, 3 (1996), 66–86.Google Scholar
- Edstem. 2023. Edstem. https://edstem.org/Google Scholar
- David C. Funder and Daniel J. Ozer. 2019. Evaluating Effect Size in Psychological Research: Sense and Nonsense. Advances in Methods and Practices in Psychological Science 2, 2 (2019), 156–168. https://doi.org/10.1177/2515245919847202Google ScholarCross Ref
- Chris M. Golde. 2008. Applying lessons from professional education to the preparation of the professoriate. New Directions for Teaching and Learning 2008, 113 (2008), 17–25. https://doi.org/10.1002/tl.305 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/tl.305Google ScholarCross Ref
- Gunnar Harboe, Jonas Minke, Ioana Ilea, and Elaine M. Huang. 2012. Computer Support for Collaborative Data Analysis: Augmenting Paper Affinity Diagrams. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1179–1182. https://doi.org/10.1145/2145204.2145379Google ScholarDigital Library
- Adrián Hernández-López, Ricardo Colomo-Palacios, and Ángel García-Crespo. 2012. Productivity in software engineering: A study of its meanings for practitioners: Understanding the concept under their standpoint. In 7th Iberian Conference on Information Systems and Technologies (CISTI 2012). 1–6.Google Scholar
- Matthew C. Jadud. 2006. Methods and Tools for Exploring Novice Compilation Behaviour. In Proceedings of the Second International Workshop on Computing Education Research (Canterbury, United Kingdom) (ICER ’06). Association for Computing Machinery, New York, NY, USA, 73–84. https://doi.org/10.1145/1151588.1151600Google ScholarDigital Library
- Scott D Johnson and Rita McDonough Fischbach. 1992. Teaching Problem Solving and Technical Mathematics through Cognitive Apprenticeship at the Community College Level. https://eric.ed.gov/?id=ED352455Google Scholar
- Despina Koletsi and Nikolaos Pandis. 2016. The chi-square test for trend. American Journal of Orthodontics and Dentofacial Orthopedics 150, 6 (2016), 1066–1067. https://doi.org/10.1016/j.ajodo.2016.10.001Google ScholarCross Ref
- Michael Kölling and David J. Barnes. 2004. Enhancing Apprentice-Based Learning of Java. SIGCSE Bull. 36, 1 (mar 2004), 286–290. https://doi.org/10.1145/1028174.971403Google ScholarDigital Library
- H. Chad Lane and Kurt VanLehn. 2005. Intention-Based Scoring: An Approach to Measuring Success at Solving the Composition Problem. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (St. Louis, Missouri, USA) (SIGCSE ’05). Association for Computing Machinery, New York, NY, USA, 373–377. https://doi.org/10.1145/1047344.1047471Google ScholarDigital Library
- Saskia le Cessie, Jelle J Goeman, and Olaf M Dekkers. 2020. Who is afraid of non-normal data? Choosing between parametric and non-parametric tests. European Journal of Endocrinology 182, 2 (2020), E1 – E3. https://doi.org/10.1530/EJE-19-0922Google ScholarCross Ref
- H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50 – 60. https://doi.org/10.1214/aoms/1177730491Google ScholarCross Ref
- Rifat Sabbir Mansur, Ayaan M. Kazerouni, Stephen H. Edwards, and Clifford A. Shaffer. 2020. Exploring the Bug Investigation Techniques of Intermediate Student Programmers. In Proceedings of the 20th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’20). Association for Computing Machinery, New York, NY, USA, Article 2, 10 pages. https://doi.org/10.1145/3428029.3428040Google ScholarDigital Library
- Mary McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB 22 (10 2012), 276–82. https://doi.org/10.11613/BM.2012.031Google Scholar
- Nachiappan Nagappan, Laurie Williams, Miriam Ferzli, Eric Wiebe, Kai Yang, Carol Miller, and Suzanne Balik. 2003. Improving the CS1 Experience with Pair Programming. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (Reno, Navada, USA) (SIGCSE ’03). Association for Computing Machinery, New York, NY, USA, 359–362. https://doi.org/10.1145/611892.612006Google ScholarDigital Library
- Clifford Nowell and Richard M. Alston. 2007. I Thought I Got an A! Overconfidence Across the Economics Curriculum. The Journal of Economic Education 38, 2 (2007), 131–142. https://doi.org/10.3200/JECE.38.2.131-142 arXiv:https://doi.org/10.3200/JECE.38.2.131-142Google ScholarCross Ref
- John Paxton. 2002. Live Programming as a Lecture Technique. J. Comput. Sci. Coll. 18, 2 (dec 2002), 51–56.Google ScholarDigital Library
- Harry O. Posten. 1984. Robustness of the Two-Sample T-Test. In Robustness of Statistical Methods and Nonparametric Statistics, Dieter Rasch and Moti Lal Tiku (Eds.). Springer Netherlands, Dordrecht, 92–99. https://doi.org/10.1007/978-94-009-6528-7_23Google Scholar
- Adalbert Gerald Soosai Raj, Pan Gu, Eda Zhang, Arokia Xavier Annie R, Jim Williams, Richard Halverson, and Jignesh M. Patel. 2020. Live-Coding vs Static Code Examples: Which is Better with Respect to Student Learning and Cognitive Load?. In Proceedings of the Twenty-Second Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE’20). Association for Computing Machinery, New York, NY, USA, 152–159. https://doi.org/10.1145/3373165.3373182Google ScholarDigital Library
- Adalbert Gerald Soosai Raj, Jignesh M. Patel, Richard Halverson, and Erica Rosenfeld Halverson. 2018. Role of Live-Coding in Learning Introductory Programming. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’18). Association for Computing Machinery, New York, NY, USA, Article 13, 8 pages. https://doi.org/10.1145/3279720.3279725Google ScholarDigital Library
- Robert S. Rist. 1990. Variability in program design: the interaction of process with knowledge. International Journal of Man-Machine Studies 33, 3 (1990), 305–322. https://doi.org/10.1016/S0020-7373(05)80121-XGoogle ScholarDigital Library
- Robert S. Rist. 1995. Program structure and design. Cognitive Science 19, 4 (1995), 507–561. https://doi.org/10.1016/0364-0213(95)90009-8Google ScholarCross Ref
- Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and Teaching Programming: A Review and Discussion. Computer Science Education 13, 2 (2003), 137–172. https://doi.org/10.1076/csed.13.2.137.14200 arXiv:https://doi.org/10.1076/csed.13.2.137.14200Google ScholarCross Ref
- Marc J. Rubin. 2013. The Effectiveness of Live-Coding to Teach Introductory Programming. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 651–656. https://doi.org/10.1145/2445196.2445388Google ScholarDigital Library
- Randall E. Schumacker. 2023. Learning Statistics Using R. SAGE Publications, Inc., 55 City Road, London, Chapter 12. https://doi.org/10.4135/9781506300160Google Scholar
- Ana Selvaraj, Eda Zhang, Leo Porter, and Adalbert Gerald Soosai Raj. 2021. Live Coding: A Review of the Literature. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 (Virtual Event, Germany) (ITiCSE ’21). Association for Computing Machinery, New York, NY, USA, 164–170. https://doi.org/10.1145/3430665.3456382Google ScholarDigital Library
- Anshul Shah, Vardhan Agarwal, Michael Granado, John Driscoll, Emma Hogan, Leo Porter, William Griswold, and Adalbert Gerald Soosai Raj. 2023. The Impact of a Remote Live-Coding Pedagogy on Student Programming Processes, Grades, and Lecture Questions Asked. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V.1 (Turku, Finland) (ITiCSE ’23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3587102.3588846Google ScholarDigital Library
- Anshul Shah, Michael Granado, Mrinal Sharma, John Driscoll, Leo Porter, William Griswold, and Adalbert Gerald Soosai Raj. 2023. Understanding and Measuring Incremental Development in CS1. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (Toronto, ON, Canada) (SIGCSE ’23). Association for Computing Machinery, New York, NY, USA, 7 pages. https://doi.org/10.1145/3545945.3569880Google ScholarDigital Library
- Amy Shannon and Valerie Summet. 2015. Live Coding in Introductory Computer Science Courses. J. Comput. Sci. Coll. 31, 2 (dec 2015), 158–164.Google Scholar
- E. Soloway. 1986. Learning to Program = Learning to Construct Mechanisms and Explanations. Commun. ACM 29, 9 (sep 1986), 850–858. https://doi.org/10.1145/6592.6594Google ScholarDigital Library
- Stepik. 2023. Stepik. https://stepik.org/course/125129/Google Scholar
- Razieh Tadayon Nabavi. 2012. Bandura’s Social Learning Theory & Social Cognitive Learning Theory by Razieh Tadayon Nabavi and Mohammad Sadegh Bijandi. https://www.researchgate.net/publication/267750204_Bandura’s_Social_Learning_Theory_Social_Cognitive_Learning_TheoryGoogle Scholar
- Sheng-Rong Tan, Yu-Tzu Lin, and Jia-Sin Liou. 2016. Teaching by demonstration: programming instruction by using live-coding videos. In Proceedings of EdMedia + Innovate Learning 2016. Association for the Advancement of Computing in Education (AACE), Vancouver, BC, Canada, 1294–1298. https://www.learntechlib.org/p/173121Google Scholar
- Anne Venables, Grace Tan, and Raymond Lister. 2009. A Closer Look at Tracing, Explaining and Code Writing Skills in the Novice Programmer. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop (Berkeley, CA, USA) (ICER ’09). Association for Computing Machinery, New York, NY, USA, 117–128. https://doi.org/10.1145/1584322.1584336Google ScholarDigital Library
- Arto Vihavainen, Matti Paksula, and Matti Luukkainen. 2011. Extreme Apprenticeship Method in Teaching Programming for Beginners. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE ’11). Association for Computing Machinery, New York, NY, USA, 93–98. https://doi.org/10.1145/1953163.1953196Google ScholarDigital Library
- Maureen M. Villamor. 2020. A Review on Process-oriented Approaches for Analyzing Novice Solutions to Programming Problems. Research and Practice in Technology Enhanced Learning 15, 1 (Apr 2020), 8. https://doi.org/10.1186/s41039-020-00130-yGoogle ScholarCross Ref
- Christopher Watson, Frederick W.B. Li, and Jamie L. Godwin. 2013. Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior. In 2013 IEEE 13th International Conference on Advanced Learning Technologies. 319–323. https://doi.org/10.1109/ICALT.2013.99Google ScholarDigital Library
- Donald W. Zimmerman. 1987. Comparative Power of Student T Test and Mann-Whitney U Test for Unequal Sample Sizes and Variances. The Journal of Experimental Education 55, 3 (1987), 171–174. http://www.jstor.org/stable/20151691Google ScholarCross Ref
Index Terms
- An Empirical Evaluation of Live Coding in CS1
Recommendations
The Impact of a Remote Live-Coding Pedagogy on Student Programming Processes, Grades, and Lecture Questions Asked
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1Live coding---a pedagogical technique in which an instructor plans, writes, and executes code in front of a class---is generally considered a best practice when teaching programming. However, only a few studies have evaluated the effect of live coding ...
The experience of using Moodle and Elluminate Live! as learning tools in the Hashemite University case study: computer skills course
In Jordan, e-learning tools are used to enhance the traditional learning process in traditional class rooms. Course management systems CMS are being utilised by many Jordanian universities to improve the learning process. The Hashemite University ...
Examining the use of prompts to facilitate self-regulated learning in Massive Open Online Courses
AbstractThe limited instructional support in Massive Open Online Courses (MOOCs) inherently demands learners to self-regulate their learning. MOOC research shows that learners are more successful when they engage in self-regulated learning (...
Highlights- Prompts supporting self-regulated learning (SRL) were examined in MOOCs.
- SRL ...
Comments