ABSTRACT
Background and context. “Explain in Plain English” (EiPE) questions ask students to explain the high-level purpose of code, requiring them to understand the macrostructure of the program’s intent. A lot is known about techniques that experts use to comprehend code, but less is known about how we should teach novices to develop this capability.
Objective. Identify techniques that can be taught to students to assist them in developing their ability to comprehend code and contribute to the body of knowledge of how novices develop their code comprehension skills.
Method. We developed interventions that could be taught to novices motivated by previous research about how experts comprehend code: prompting students to identify beacons, identify the role of variables, tracing, and abstract tracing. We conducted think-aloud interviews of introductory programming students solving EiPE questions, varying which interventions each student was taught. Some participants were interviewed multiple times throughout the semester to observe any changes in behavior over time.
Findings. Identifying beacons and the name of variable roles were rarely helpful, as they did not encourage students to integrate their understanding of that piece in relation to other lines of code. However, prompting students to explain each variable’s purpose helped them focus on useful subsets of the code, which helped manage cognitive load. Tracing was helpful when students incorrectly recognized common programming patterns or made mistakes comprehending syntax (text-surface). Prompting students to pick inputs that potentially contradicted their current understanding of the code was found to be a simple approach to them effectively selecting inputs to trace. Abstract tracing helped students see high-level, functional relationships between variables. In addition, we observed student spontaneously sketching algorithmic visualizations that similarly helped them see relationships between variables.
Implications. Because students can get stuck at many points in the process of code comprehension, there seems to be no silver bullet technique that helps in every circumstance. Instead, effective instruction for code comprehension will likely involve teaching a collection of techniques. In addition to these techniques, meta-knowledge about when to apply each technique will need to be learned, but that is left for future research. At present, we recommend teaching a bottom-up, concrete-to-abstract approach.
- Mordechai Ben-Ari and Jorma Sajaniemi. 2004. Roles of variables as seen by CS educators. ACM Sigcse Bulletin 36, 3 (2004), 52–56.Google ScholarDigital Library
- Jens Bennedsen and Carsten Schulte. 2010. BlueJ visual debugger for learning the execution of object-oriented programs?ACM Transactions on Computing Education (TOCE) 10, 2 (2010), 1–22.Google Scholar
- Sarah Blankenship. 2022. Learning to Reason About Code with Assertions: An Exploration with Two Student Populations. (2022).Google Scholar
- Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International journal of man-machine studies 18, 6 (1983), 543–554.Google ScholarCross Ref
- Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 255–265.Google ScholarDigital Library
- Pauli Byckling and Jorma Sajaniemi. 2006. Roles of variables and programming skills improvement. ACM SIGCSE Bulletin 38, 1 (2006), 413–417.Google ScholarDigital Library
- John H Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry.American psychologist 34, 10 (1979), 906.Google Scholar
- Megan Fowler, Jason Hallstrom, Joseph Hollingsworth, Eileen Kraemer, Murali Sitaraman, Yu-Shan Sun, Jiadi Wang, and Gloria Washington. 2021. Tool-Aided Learning of Code Reasoning with Abstraction in the CS Curriculum. Informatics in Education 20, 4 (2021).Google Scholar
- Max Fowler, David H Smith IV, Mohammed Hassan, Seth Poulsen, Matthew West, and Craig Zilles. 2022. Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication study. Computer Science Education (2022), 1–29.Google Scholar
- Edward M Gellenbeck and Curtis R Cook. 1991. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop. Ablex Publishing, Norwood, NJ, 65–81.Google Scholar
- David Ginat. 2001. Misleading intuition in algorithmic problem solving. ACM SIGCSE Bulletin 33, 1 (2001), 21–25.Google ScholarDigital Library
- Susan Goldin-Meadow. 1999. The role of gesture in communication and thinking. Trends in cognitive sciences 3, 11 (1999), 419–429.Google Scholar
- Alex Groce, Iftekhar Ahmed, Carlos Jensen, Paul E McKenney, and Josie Holmes. 2018. How verified (or tested) is my code? Falsification-driven verification and testing. Automated Software Engineering 25 (2018), 917–960.Google ScholarDigital Library
- Leo Gugerty and Gary Olson. 1986. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems. 171–174.Google ScholarDigital Library
- Mohammed Hassan and Craig Zilles. 2023. On Students’ Usage of Tracing for Understanding Code. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 129–136.Google ScholarDigital Library
- Vighnesh Iyer and Craig Zilles. 2021. Pattern Census: A Characterization of Pattern Usage in Early Programming Courses. In Proceedings of the SIGCSE Technical Symposium (SIGCSE).Google ScholarDigital Library
- Cruz Izu, Cheryl Pope, and Amali Weerasinghe. 2017. On the ability to reason about program behaviour: A think-aloud study. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education. 305–310.Google ScholarDigital Library
- Cruz Izu, Carsten Schulte, Ashish Aggarwal, Quintin Cutts, Rodrigo Duran, Mirela Gutica, Birte Heinemann, Eileen Kraemer, Violetta Lonati, Claudio Mirolo, 2019. Fostering program comprehension in novice programmers-learning activities and learning trajectories. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education. 27–52.Google ScholarDigital Library
- Philipp Kather, Rodrigo Duran, and Jan Vahrenhold. 2021. Through (tracking) their eyes: Abstraction and complexity in program comprehension. ACM Transactions on Computing Education (TOCE) 22, 2 (2021), 1–33.Google Scholar
- Marja Kuittinen and Jorma Sajaniemi. 2004. Teaching roles of variables in elementary programming courses. In Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education. 57–61.Google ScholarDigital Library
- S Letovsky. 1986. Cognitive Processes in Program Comprehension: First Workshop. E. Soloway and S. Iyengar eds.Google ScholarDigital Library
- RF Lister. 2007. The neglected middle novice programmer: Reading and writing without abstracting. National Advisory Committee on Computing Qualifications (2007).Google Scholar
- Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further Evidence of a Relationship Between Explaining, Tracing and Writing Skills in Introductory Programming. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (Paris, France) (ITiCSE ’09). ACM, New York, NY, USA, 161–165. https://doi.org/10.1145/1562877.1562930Google ScholarDigital Library
- David C Littman, Jeannine Pinto, Stanley Letovsky, and Elliot Soloway. 1987. Mental models and software maintenance. Journal of Systems and Software 7, 4 (1987), 341–355.Google ScholarDigital Library
- Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the Fourth International Workshop on Computing Education Research. ACM, 101–112.Google ScholarDigital Library
- Jean M Mandler and Felice Orlich. 1993. Analogical transfer: The roles of schema abstraction and awareness. Bulletin of the Psychonomic Society 31, 5 (1993), 485–487.Google ScholarCross Ref
- Lauren Margulieux and Richard Catrambone. 2017. Using learners’ self-explanations of subgoals to guide initial problem solving in app inventor. In Proceedings of the 2017 ACM Conference on International Computing Education Research. 21–29.Google ScholarDigital Library
- Lauren E Margulieux, Briana B Morrison, and Adrienne Decker. 2020. Reducing withdrawal and failure rates in introductory programming with subgoal labeled worked examples. International Journal of STEM Education 7 (2020), 1–16.Google ScholarCross Ref
- Michael McCracken 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (Canterbury, UK) (ITiCSE-WGR ’01). ACM, New York, NY, USA, 125–180.Google Scholar
- Greg L Nelson, Benjamin Xie, and Amy J Ko. 2017. Comprehension first: evaluating a novel pedagogy and tutoring system for program tracing in CS1. In Proceedings of the 2017 ACM conference on international computing education research. 2–11.Google ScholarDigital Library
- Michael P O’Brien, Jim Buckley, and Teresa M Shaft. 2004. Expectation-based, inference-based, and bottom-up software comprehension. Journal of Software Maintenance and Evolution: Research and Practice 16, 6 (2004), 427–447.Google ScholarDigital Library
- Nancy Pennington. 1987. Comprehension strategies in programming. In Empirical Studies of Programmers: Second Workshop, 1987. 100–113.Google ScholarDigital Library
- Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology 19, 3 (1987), 295–341.Google Scholar
- Rüdiger F Pohl. 2022. Cognitive Illusions. Cognitive Illusions: Intriguing Phenomena in Thinking, Judgment, and Memory (2022), 3.Google ScholarCross Ref
- Karl Popper. 2005. The logic of scientific discovery. Routledge.Google Scholar
- James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive difficulties faced by novice programmers in automated assessment tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 41–50.Google ScholarDigital Library
- Michael Raadt, Mark Toleman, and Richard Watson. 2007. Incorporating programming strategies explicitly into curricula. In Proceedings of the Seventh Baltic Sea Conference on Computing Education Research (Koli Calling 2007).Google Scholar
- Nikolaus Ritt. 2005. Analogy and Transfer: Encoding the Problem at the Right Level of Abstraction. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 27.Google Scholar
- Jorma Sajaniemi. 2002. An empirical analysis of roles of variables in novice-level procedural programs. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. IEEE, 37–39.Google ScholarCross Ref
- Jorma Sajaniemi, Mordechai Ben-Ari, Pauli Byckling, Petri Gerdt, and Yevgeniya Kulikova. 2006. Roles of variables in three programming paradigms. Computer Science Education 16, 4 (2006), 261–279.Google ScholarCross Ref
- Jorma Sajaniemi and Marja Kuittinen. 2003. Program animation based on the roles of variables. In Proceedings of the 2003 ACM symposium on Software visualization. 7–ff.Google ScholarDigital Library
- Jorma Sajaniemi and Marja Kuittinen. 2005. An experiment on using roles of variables in teaching introductory programming. Computer Science Education 15, 1 (2005), 59–82.Google ScholarCross Ref
- Emmanuel Sander and Jean-François Richard. 1997. Analogical transfer as guided by an abstraction process: The case of learning by doing in text editing.Journal of experimental psychology: learning, memory, and cognition 23, 6 (1997), 1459.Google Scholar
- Carsten Schulte. 2008. Block Model: an educational model of program comprehension as a tool for a scholarly approach to teaching. In Proceedings of the Fourth international Workshop on Computing Education Research. 149–160.Google ScholarDigital Library
- Daniel L Schwartz. 1993. The construction and analogical transfer of symbolic visualizations. Journal of research in science teaching 30, 10 (1993), 1309–1325.Google ScholarCross Ref
- Nianfeng Shi. 2021. Improving undergraduate novice programmer comprehension through case-based teaching with roles of variables to provide scaffolding. Information 12, 10 (2021), 424.Google ScholarCross Ref
- Shuhaida Shuhidan, Margaret Hamilton, and Daryl D’souza. 2009. A taxonomic study of novice programming summative assessment. In Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95. Citeseer, 147–156.Google ScholarDigital Library
- Elliot Soloway. 1986. Learning to program= learning to construct mechanisms and explanations. Commun. ACM 29, 9 (1986), 850–858.Google ScholarDigital Library
- Elliot Soloway, Beth Adelson, and Kate Ehrlich. 1988. Knowledge and processes in the comprehension of computer programs. The nature of expertise (1988), 129–152.Google Scholar
- Juha Sorva 2012. Visual program simulation in introductory programming education. Aalto University.Google Scholar
- Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A review of generic program visualization systems for introductory programming education. ACM Transactions on Computing Education (TOCE) 13, 4 (2013), 1–64.Google ScholarDigital Library
- James C Spohrer and Elliot Soloway. 1986. Novice mistakes: Are the folk wisdoms correct?Commun. ACM 29, 7 (1986), 624–632.Google ScholarDigital Library
- James C Spohrer, Elliot Soloway, and Edgar Pope. 1985. A goal/plan analysis of buggy Pascal programs. Human–Computer Interaction 1, 2 (1985), 163–207.Google Scholar
- Donna Teague. 2015. Neo-Piagetian theory and the novice programmer. Diss. Queensland University of Technology (2015).Google Scholar
- Donna Teague, Malcolm Corney, Alireza Ahadi, and Raymond Lister. 2013. A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers. In Proceedings of the 15th Australasian Computing Education Conference [Conferences in Research and Practice in Information Technology, Volume 136]. Australian Computer Society, 87–95.Google Scholar
- Donna Teague and Raymond Lister. 2014. Blinded by their Plight: Tracing and the Preoperational Programmer.. In PPIG. 8.Google Scholar
- Donna Teague and Raymond Lister. 2014. Longitudinal think aloud study of a novice programmer. In Conferences in Research and Practice in Information Technology Series.Google Scholar
- Vesa Vainio and Jorma Sajaniemi. 2007. Factors in novice programmers’ poor tracing skills. ACM SIGCSE Bulletin 39, 3 (2007), 236–240.Google ScholarDigital Library
- Iris Vessey. 1985. Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies 23, 5 (1985), 459–494.Google ScholarCross Ref
- Anneliese Von Mayrhauser and A Marie Vans. 1995. Program comprehension during software maintenance and evolution. Computer 28, 8 (1995), 44–55.Google ScholarDigital Library
- Renske Weeda, Sjaak Smetsers, and Erik Barendsen. 2023. Unraveling novices’ code composition difficulties. Computer Science Education (2023), 1–28.Google Scholar
- Mark Weiser. 1984. Program Slicing. IEEE Transactions on Software Engineering SE-10, 4 (1984), 352–357. https://doi.org/10.1109/TSE.1984.5010248Google ScholarDigital Library
- JL Whalley, Tony Clear, Phil Robbins, and Errol Thompson. 2011. Salient elements in novice solutions to code writing problems. (2011).Google Scholar
- Susan Wiedenbeck. 1991. The initial stage of program comprehension. International Journal of Man-Machine Studies 35, 4 (1991), 517–540.Google ScholarDigital Library
- Susan Wiedenbeck and Jean Scholtz. 1989. Beacons: A knowledge structure in program comprehension. In Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.). 82–87.Google Scholar
- Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Andrew J Ko. 2019. A theory of instruction for introductory programming skills. Computer Science Education 29, 2-3 (2019), 205–253.Google ScholarCross Ref
Index Terms
- Evaluating Beacons, the Role of Variables, Tracing, and Abstract Tracing for Teaching Novices to Understand Program Intent
Recommendations
On Students' Usage of Tracing for Understanding Code
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1Explain in Plain English (EiPE) questions evaluate whether students can understand and explain the high-level purpose of code. We conducted a qualitative think-aloud study of introductory programming students solving EiPE questions. In this paper, we ...
The Effect of Sketching and Tracing on Instructors' Understanding of Student Misconceptions
ICER '17: Proceedings of the 2017 ACM Conference on International Computing Education ResearchThe operation of the notional machine presents a hidden but crucial process in students' understanding of introductory programming. When students trace though code, simulating the operation of the notional machine, this hidden operation becomes evident. ...
The Role of Working Memory in Program Tracing
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsProgram tracing, or mentally simulating a program on concrete inputs, is an important part of general program comprehension. Programs involve many kinds of virtual state that must be held in memory, such as variable/value pairs and a call stack. In ...
Comments