skip to main content
10.1145/3632620.3671120acmconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article
Open access

Validating, Refining, and Identifying Programming Plans Using Learning Curve Analysis on Code Writing Data

Published: 12 August 2024 Publication History

Abstract

Background and Context: A major difference between expert and novice programmers is the ability to recognize and apply common and meaningful patterns in code. Previous works have attempted to identify these patterns as programming plans, such as counting or filtering the items of a collection. However, these efforts primarily relied on expert opinions and yielded many varied sets of plans. No methods have been applied to evaluate these various programming plans as far as their alignment with novices’ cognitive development.
Objectives: In this work, we investigate which programming plans are learned as discrete skills. To this end, we evaluate how well students transfer their knowledge between problems that test a particular plan. Further, we explore how plan definitions can be improved to better represent student cognition using historical data on student performance collected from a programming course.
Method: We apply learning curve analysis, a method for modeling student improvement on problems that test a particular skill, using programming plans as a skill model. More specifically, we study student submissions on code-writing exercises in Python from a CS1 class for non-majors that includes many small programming problems as well as implicit and explicit instruction on patterns. We compare the learning curves for ten programming plans across seven semesters of the same course.
Findings: Students develop the skill of using some programming plans in their solutions, indicated by consistent declines in error rates on practice opportunities for a subset of plans in multiple semesters with various conditions. Most consistently learned plans have clear and concrete goals that can be explained in natural language, as opposed to having abstract definitions or being explained in terms of language structures.
Implications: We show that learning curve analysis can be used to empirically assess the cognitive validity of proposed programming plans, as well as compare various plan models. However, our work also indicates that instructors should be cautious when assuming that introductory programming students can apply more abstract programming plans to successfully solve new problems, as plans with increased specificity tend to better explain the learning process in our observations.

References

[1]
Vincent Aleven and Kenneth R. Koedinger. 2013. Knowledge Component (KC) Approaches to Learner Modeling. Design recommendations for intelligent tutoring systems 1 (2013), 165–182.
[2]
Isaac Alpizar-Chacon, Sergey Sosnovsky, and Peter Brusilovsky. 2023. Measuring the Quality of Domain Models Extracted from Textbooks with Learning Curves Analysis. In Artificial Intelligence in Education, Ning Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, and Vania Dimitrova (Eds.). Springer Nature Switzerland, Cham, 804–809. https://doi.org/10.1007/978-3-031-36272-9_75
[3]
John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger, and Ray. Pelletier. 1995. Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences 4, 2 (April 1995), 167–207. https://doi.org/10.1207/s15327809jls0402_2
[4]
Owen Astrachan, Garrett Mitchener, Geoffrey Berry, and Landon Cox. 1998. Design Patterns: An Essential Component of CS Curricula. In Proceedings of the Twenty-Ninth SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’98). Association for Computing Machinery, New York, NY, USA, 153–160. https://doi.org/10.1145/273133.273182
[5]
Owen Astrachan and Eugene Wallingford. 1998. Loop Patterns. In Proceedings of the Fifth Pattern Languages of Programs Conference.
[6]
Robert L Bangert-Drowns, James A Kulik, and Chen-Lin C Kulik. 1991. Effects of frequent classroom testing. The journal of educational research 85, 2 (1991), 89–99.
[7]
Joseph Bergin. 1999. Patterns for Selection. https://csis.pace.edu/~bergin/patterns/Patternsv4.html.
[8]
Paul D. Bliese, David Chan, and Robert E. Ployhart. 2007. Multilevel Methods: Future Directions in Measurement, Longitudinal Analyses, and Nonnormal Outcomes. Organizational Research Methods 10, 4 (2007), 551–563. https://doi.org/10.1177/1094428107301102
[9]
John Bransford, Ted Hasselbring, Brigid Barron, Stan Kulewicz, Joan Littlefield, and Laura Goin. 1988. Uses of macro-contexts to facilitate mathematical thinking. The teaching and assessing of mathematical problem solving 3 (1988), 125–147.
[10]
Hao Cen, Kenneth Koedinger, and Brian Junker. 2006. Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement. In Intelligent Tutoring Systems(Lecture Notes in Computer Science), Mitsuru Ikeda, Kevin D. Ashley, and Tak-Wai Chan (Eds.). Springer, Berlin, Heidelberg, 164–175. https://doi.org/10.1007/11774303_17
[11]
Michelene T. H. Chi, Robert E. Glaser, and Marshall J. Farr. 1988. The Nature of Expertise. https://api.semanticscholar.org/CorpusID:143028015
[12]
Michael J Clancy. 1996. Designing Pascal Solutions: Case studies using data structures. WH Freeman & Co.
[13]
Kathryn Cunningham, Barbara J. Ericson, Rahul Agrawal Bejarano, and Mark Guzdial. 2021. Avoiding the Turing Tarpit: Learning Conversational Programming by Starting from Code’s Purpose. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–15. https://doi.org/10.1145/3411764.3445571
[14]
Michael de Raadt, Richard Watson, and Mark Toleman. 2009. Teaching and Assessing Programming Strategies Explicitly. In Proceedings of the 11th Australasian Computing Education Conference (ACE 2009), Margaret Hamilton and Tony Clear (Eds.). Vol. 95. Bedford Park, South Australia, 45–54.
[15]
J. Philip East, S. Rebecca Thomas, Eugene Wallingford, Walter Beck, and Janet Drake. 1996. Pattern Based Programming Instruction. In 1996 Annual Conference. 1.349.1–1.349.10.
[16]
Kathi Fisler, Shriram Krishnamurthi, and Janet Siegmund. 2016. Modernizing Plan-Composition Studies. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM, Memphis Tennessee USA, 211–216. https://doi.org/10.1145/2839509.2844556
[17]
Max Fowler and Craig Zilles. 2021. Superficial Code-Guise: Investigating the Impact of Surface Feature Changes on Students’ Programming Question Scores. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 3–9. https://doi.org/10.1145/3408877.3432413
[18]
D. J. Gilmore and T. R. G. Green. 1988. Programming Plans and Programming Expertise. The Quarterly Journal of Experimental Psychology Section A 40, 3 (Aug. 1988), 423–442. https://doi.org/10.1080/02724988843000005
[19]
David Ginat. 2009. Interleaved pattern composition and scaffolded learning. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (Paris, France) (ITiCSE ’09). Association for Computing Machinery, New York, NY, USA, 109–113. https://doi.org/10.1145/1562877.1562915
[20]
David Ginat, Eti Menashe, and Amal Taya. 2013. Novice Difficulties with Interleaved Pattern Composition. In Informatics in Schools. Sustainable Informatics Education for Pupils of all Ages, Ira Diethelm and Roland T. Mittermeir (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 57–67.
[21]
B.G. Glaser. 1992. Emergence Vs Forcing: Basics of Grounded Theory Analysis. Sociology Press. https://books.google.com/books?id=1ZJiQgAACAAJ
[22]
Cyril Goutte and Guillaume Durand. 2020. Confident Learning Curves in Additive Factors Modeling.International Educational Data Mining Society (2020).
[23]
Vighnesh Iyer and Craig Zilles. 2021. Pattern Census: A Characterization of Pattern Usage in Early Programming Courses. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education(SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3408877.3432442
[24]
Vighnesh Narayanan Iyer. 2020. An Analysis of Pattern Usage in CS1 Courses. Thesis. University of Illinois at Urbana-Champaign.
[25]
Cruz Izu, Violetta Lonati, Anna Morpurgo, and Mario Sanchez. 2021. An Inventory of Goals from CS1 Programs Processing a Data Series. In 2021 IEEE Frontiers in Education Conference (FIE). 1–8. https://doi.org/10.1109/FIE49875.2021.9637360
[26]
Philipp Kather, Rodrigo Duran, and Jan Vahrenhold. 2021. Through (tracking) their eyes: Abstraction and complexity in program comprehension. ACM Transactions on Computing Education (TOCE) 22, 2 (2021), 1–33.
[27]
Kenneth R. Koedinger, Ryan Sjd Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010. A Data Repository for the EDM Community: The PSLC DataShop. Handbook of educational data mining 43 (2010), 43–56.
[28]
Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning. Cognitive Science 36, 5 (2012), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
[29]
Kenneth R. Koedinger, Elizabeth A. McLaughlin, and John C. Stamper. 2012. Automated Student Model Improvement. Technical Report. International Educational Data Mining Society.
[30]
Kenneth R. Koedinger, John C. Stamper, Elizabeth A. McLaughlin, and Tristan Nixon. 2013. Using Data-Driven Discovery of Better Student Models to Improve Student Learning. In Artificial Intelligence in Education, H. Chad Lane, Kalina Yacef, Jack Mostow, and Philip Pavlik (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 421–430.
[31]
Klaus Krippendorff. 2011. Computing Krippendorff’s Alpha-Reliability. 43 (Jan. 2011). https://repository.upenn.edu/handle/20.500.14332/2089
[32]
Marja Kuittinen and Jorma Sajaniemi. 2004. Teaching roles of variables in elementary programming courses. In Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (Leeds, United Kingdom) (ITiCSE ’04). Association for Computing Machinery, New York, NY, USA, 57–61. https://doi.org/10.1145/1007996.1008014
[33]
Filip Lievens and Paul R. Sackett. 2007. Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms.Journal of Applied Psychology 92 (2007), 1043–1055. https://doi.org/10.1037/0021-9010.92.4.1043
[34]
Brent Martin, Antonija Mitrovic, Kenneth R. Koedinger, and Santosh Mathan. 2011. Evaluating and Improving Adaptive Educational Systems with Learning Curves. User Modeling and User-Adapted Interaction 21, 3 (Aug. 2011), 249–283. https://doi.org/10.1007/s11257-010-9084-2
[35]
VT Mawhinney, DE Bostow, DR Laws, GJ Blumenfeld, and BL Hopkins. 1971. A comparison of students studying-behavior produced by daily, weekly, and three-week testing schedules. Journal of Applied Behavior Analysis 4, 4 (1971), 257–264.
[36]
Orna Muller. 2005. Pattern Oriented Instruction and the Enhancement of Analogical Reasoning. In Proceedings of the First International Workshop on Computing Education Research(ICER ’05). Association for Computing Machinery, New York, NY, USA, 57–67. https://doi.org/10.1145/1089786.1089792
[37]
Orna Muller, David Ginat, and Bruria Haberman. 2007. Pattern-Oriented Instruction and Its Influence on Problem Decomposition and Solution Construction. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education(ITiCSE ’07). Association for Computing Machinery, New York, NY, USA, 151–155. https://doi.org/10.1145/1268784.1268830
[38]
Mitchell J. Nathan, Kenneth R. Koedinger, and Martha W. Alibali. 2001. Expert Blind Spot: When Content Knowledge Eclipses Pedagogical Content Knowledge. In Proceedings of the Third International Conference on Cognitive Science, Vol. 644648. 644–648.
[39]
Allen Newell and Paul S. Rosenbloom. 2013. Mechanisms of Skill Acquisition and the Law of Practice. In Cognitive Skills and Their Acquisition. Psychology Press, 1–55.
[40]
Huy Nguyen, Yeyu Wang, John Stamper, and Bruce M. McLaren. 2019. Using Knowledge Component Modeling to Increase Domain Understanding in a Digital Learning Game. Technical Report. International Educational Data Mining Society.
[41]
Peterson K Ozili. 2023. The acceptable R-square in empirical modelling for social science research. In Social research methodology and publishing results. IGI global, 134–143.
[42]
Rebecca Passonneau. 2006. Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, and Daniel Tapias (Eds.). European Language Resources Association (ELRA), Genoa, Italy.
[43]
Jihyun Rho, Martina Rau, and Barry Vanveen. 2022. Investigating Growth of Representational Competencies by Knowledge-Component Model. In Proceedings of the 15th International Conference on Educational Data Mining. 346. https://doi.org/10.5281/zenodo.6853077
[44]
Robert S. Rist. 1989. Schema Creation in Programming. Cognitive Science 13, 3 (July 1989), 389–414. https://doi.org/10.1016/0364-0213(89)90018-9
[45]
Kelly Rivers, Erik Harpstead, and Ken Koedinger. 2016. Learning Curve Analysis for Programming: Which Concepts Do Students Struggle With?. In Proceedings of the 2016 ACM Conference on International Computing Education Research(ICER ’16). Association for Computing Machinery, New York, NY, USA, 143–151. https://doi.org/10.1145/2960310.2960333
[46]
Anthony V. Robins. 2019. Novice Programmers and Introductory Programming. In The Cambridge Handbook of Computing Education Research, Sally A. Fincher and Anthony V.Editors Robins (Eds.). Cambridge University Press, 327–376. https://doi.org/10.1017/9781108654555.013
[47]
Anthony V. Robins, Lauren E. Margulieux, and Briana B. Morrison. 2019. Cognitive Sciences for Computing Education. In The Cambridge Handbook of Computing Education Research, Sally A. Fincher and Anthony V.Editors Robins (Eds.). Cambridge University Press, 231–275. https://doi.org/10.1017/9781108654555.010
[48]
Donna Sabers, Katherine Cushing, and David Berliner. 1991. Differences Among Teachers in a Task Characterized by Simultaneity, Multidimensional, and Immediacy. American Educational Research Journal - AMER EDUC RES J 28 (03 1991), 63–88. https://doi.org/10.3102/00028312028001063
[49]
J. Sajaniemi. 2002. An empirical analysis of roles of variables in novice-level procedural programs. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. 37–39. https://doi.org/10.1109/HCC.2002.1046340
[50]
Jorma Sajaniemi and Marja Kuittinen. 2005. An Experiment on Using Roles of Variables in Teaching Introductory Programming. Computer Science Education 15, 1 (2005), 59–82. https://doi.org/10.1080/08993400500056563
[51]
Yang Shi, Robin Schmucker, Min Chi, Tiffany Barnes, and Thomas Price. 2023. KC-Finder: Automated Knowledge Component Discovery for Programming Problems. International Educational Data Mining Society (2023).
[52]
Herbert A. Simon. 1980. Problem solving and education: Issues in teaching and research. Erlbaum, Hillsdale, NJ, Chapter 6, 81–96.
[53]
Mark K. Singley and John R. Anderson. 1989. The Transfer of Cognitive Skill. https://api.semanticscholar.org/CorpusID:60731940
[54]
Elliot Soloway. 1985. From Problems to Programs via Plans: The Content and Structure of Knowledge for Introductory LISP Programming. Journal of Educational Computing Research 1, 2 (May 1985), 157–172. https://doi.org/10.2190/WK8C-BYCF-VQ5C-E307
[55]
E. Soloway. 1986. Learning to Program = Learning to Construct Mechanisms and Explanations. Commun. ACM 29, 9 (Sept. 1986), 850–858. https://doi.org/10.1145/6592.6594
[56]
Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering SE-10, 5 (Sept. 1984), 595–609. https://doi.org/10.1109/TSE.1984.5010283
[57]
Elliot M. Soloway and Beverly Woolf. 1980. Problems, Plans, and Programs. ACM SIGCSE Bulletin 12, 1 (Feb. 1980), 16–24. https://doi.org/10.1145/953032.804605
[58]
Juha Sorva, Ville Karavirta, and Ari Korhonen. 2007. Roles of variables in teaching. Journal of Information Technology Education: Research 6, 1 (2007), 407–423.
[59]
James C Spohrer, Elliot Soloway, and Edgar Pope. 1985. A Goal/Plan Analysis of Buggy Pascal Programs. Human–Computer Interaction 1, 2 (June 1985), 163–207. https://doi.org/10.1207/s15327051hci0102_4
[60]
James G. Spohrer and Elliot Soloway. 1986. Analyzing the high frequency bugs in novice programs. In Empirical Studies of Programmers Workshop, Elliot Soloway and S. Iyengar (Eds.). Ablex, 230–251.
[61]
John C. Stamper and Kenneth R. Koedinger. 2011. Human-Machine Student Model Discovery and Improvement Using DataShop. In Artificial Intelligence in Education, Gautam Biswas, Susan Bull, Judy Kay, and Antonija Mitrovic (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 353–360.
[62]
Kurt VanLehn. 2006. The behavior of tutoring systems. International journal of artificial intelligence in education 16, 3 (2006), 227–265.
[63]
Matthew West, Geoffrey L. Herman, and Craig Zilles. 2015. PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning. In 2015 ASEE Annual Conference & Exposition. 26–1238.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICER '24: Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
August 2024
539 pages
ISBN:9798400704758
DOI:10.1145/3632620
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2024

Check for updates

Author Tags

  1. CS1
  2. knowledge components
  3. learning curve analysis
  4. programming patterns
  5. programming plans

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICER 2024
Sponsor:

Acceptance Rates

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025
ACM Conference on International Computing Education Research
August 3 - 6, 2025
Charlottesville , VA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 157
    Total Downloads
  • Downloads (Last 12 months)157
  • Downloads (Last 6 weeks)51
Reflects downloads up to 31 Dec 2024

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media