skip to main content
10.1145/3568813.3600140acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article
Public Access

Evaluating Beacons, the Role of Variables, Tracing, and Abstract Tracing for Teaching Novices to Understand Program Intent

Published: 10 September 2023 Publication History

Abstract

Background and context. “Explain in Plain English” (EiPE) questions ask students to explain the high-level purpose of code, requiring them to understand the macrostructure of the program’s intent. A lot is known about techniques that experts use to comprehend code, but less is known about how we should teach novices to develop this capability.
Objective. Identify techniques that can be taught to students to assist them in developing their ability to comprehend code and contribute to the body of knowledge of how novices develop their code comprehension skills.
Method. We developed interventions that could be taught to novices motivated by previous research about how experts comprehend code: prompting students to identify beacons, identify the role of variables, tracing, and abstract tracing. We conducted think-aloud interviews of introductory programming students solving EiPE questions, varying which interventions each student was taught. Some participants were interviewed multiple times throughout the semester to observe any changes in behavior over time.
Findings. Identifying beacons and the name of variable roles were rarely helpful, as they did not encourage students to integrate their understanding of that piece in relation to other lines of code. However, prompting students to explain each variable’s purpose helped them focus on useful subsets of the code, which helped manage cognitive load. Tracing was helpful when students incorrectly recognized common programming patterns or made mistakes comprehending syntax (text-surface). Prompting students to pick inputs that potentially contradicted their current understanding of the code was found to be a simple approach to them effectively selecting inputs to trace. Abstract tracing helped students see high-level, functional relationships between variables. In addition, we observed student spontaneously sketching algorithmic visualizations that similarly helped them see relationships between variables.
Implications. Because students can get stuck at many points in the process of code comprehension, there seems to be no silver bullet technique that helps in every circumstance. Instead, effective instruction for code comprehension will likely involve teaching a collection of techniques. In addition to these techniques, meta-knowledge about when to apply each technique will need to be learned, but that is left for future research. At present, we recommend teaching a bottom-up, concrete-to-abstract approach.

References

[1]
Mordechai Ben-Ari and Jorma Sajaniemi. 2004. Roles of variables as seen by CS educators. ACM Sigcse Bulletin 36, 3 (2004), 52–56.
[2]
Jens Bennedsen and Carsten Schulte. 2010. BlueJ visual debugger for learning the execution of object-oriented programs?ACM Transactions on Computing Education (TOCE) 10, 2 (2010), 1–22.
[3]
Sarah Blankenship. 2022. Learning to Reason About Code with Assertions: An Exploration with Two Student Populations. (2022).
[4]
Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International journal of man-machine studies 18, 6 (1983), 543–554.
[5]
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 255–265.
[6]
Pauli Byckling and Jorma Sajaniemi. 2006. Roles of variables and programming skills improvement. ACM SIGCSE Bulletin 38, 1 (2006), 413–417.
[7]
John H Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry.American psychologist 34, 10 (1979), 906.
[8]
Megan Fowler, Jason Hallstrom, Joseph Hollingsworth, Eileen Kraemer, Murali Sitaraman, Yu-Shan Sun, Jiadi Wang, and Gloria Washington. 2021. Tool-Aided Learning of Code Reasoning with Abstraction in the CS Curriculum. Informatics in Education 20, 4 (2021).
[9]
Max Fowler, David H Smith IV, Mohammed Hassan, Seth Poulsen, Matthew West, and Craig Zilles. 2022. Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication study. Computer Science Education (2022), 1–29.
[10]
Edward M Gellenbeck and Curtis R Cook. 1991. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop. Ablex Publishing, Norwood, NJ, 65–81.
[11]
David Ginat. 2001. Misleading intuition in algorithmic problem solving. ACM SIGCSE Bulletin 33, 1 (2001), 21–25.
[12]
Susan Goldin-Meadow. 1999. The role of gesture in communication and thinking. Trends in cognitive sciences 3, 11 (1999), 419–429.
[13]
Alex Groce, Iftekhar Ahmed, Carlos Jensen, Paul E McKenney, and Josie Holmes. 2018. How verified (or tested) is my code? Falsification-driven verification and testing. Automated Software Engineering 25 (2018), 917–960.
[14]
Leo Gugerty and Gary Olson. 1986. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems. 171–174.
[15]
Mohammed Hassan and Craig Zilles. 2023. On Students’ Usage of Tracing for Understanding Code. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 129–136.
[16]
Vighnesh Iyer and Craig Zilles. 2021. Pattern Census: A Characterization of Pattern Usage in Early Programming Courses. In Proceedings of the SIGCSE Technical Symposium (SIGCSE).
[17]
Cruz Izu, Cheryl Pope, and Amali Weerasinghe. 2017. On the ability to reason about program behaviour: A think-aloud study. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education. 305–310.
[18]
Cruz Izu, Carsten Schulte, Ashish Aggarwal, Quintin Cutts, Rodrigo Duran, Mirela Gutica, Birte Heinemann, Eileen Kraemer, Violetta Lonati, Claudio Mirolo, 2019. Fostering program comprehension in novice programmers-learning activities and learning trajectories. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education. 27–52.
[19]
Philipp Kather, Rodrigo Duran, and Jan Vahrenhold. 2021. Through (tracking) their eyes: Abstraction and complexity in program comprehension. ACM Transactions on Computing Education (TOCE) 22, 2 (2021), 1–33.
[20]
Marja Kuittinen and Jorma Sajaniemi. 2004. Teaching roles of variables in elementary programming courses. In Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education. 57–61.
[21]
S Letovsky. 1986. Cognitive Processes in Program Comprehension: First Workshop. E. Soloway and S. Iyengar eds.
[22]
RF Lister. 2007. The neglected middle novice programmer: Reading and writing without abstracting. National Advisory Committee on Computing Qualifications (2007).
[23]
Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further Evidence of a Relationship Between Explaining, Tracing and Writing Skills in Introductory Programming. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (Paris, France) (ITiCSE ’09). ACM, New York, NY, USA, 161–165. https://doi.org/10.1145/1562877.1562930
[24]
David C Littman, Jeannine Pinto, Stanley Letovsky, and Elliot Soloway. 1987. Mental models and software maintenance. Journal of Systems and Software 7, 4 (1987), 341–355.
[25]
Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the Fourth International Workshop on Computing Education Research. ACM, 101–112.
[26]
Jean M Mandler and Felice Orlich. 1993. Analogical transfer: The roles of schema abstraction and awareness. Bulletin of the Psychonomic Society 31, 5 (1993), 485–487.
[27]
Lauren Margulieux and Richard Catrambone. 2017. Using learners’ self-explanations of subgoals to guide initial problem solving in app inventor. In Proceedings of the 2017 ACM Conference on International Computing Education Research. 21–29.
[28]
Lauren E Margulieux, Briana B Morrison, and Adrienne Decker. 2020. Reducing withdrawal and failure rates in introductory programming with subgoal labeled worked examples. International Journal of STEM Education 7 (2020), 1–16.
[29]
Michael McCracken 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (Canterbury, UK) (ITiCSE-WGR ’01). ACM, New York, NY, USA, 125–180.
[30]
Greg L Nelson, Benjamin Xie, and Amy J Ko. 2017. Comprehension first: evaluating a novel pedagogy and tutoring system for program tracing in CS1. In Proceedings of the 2017 ACM conference on international computing education research. 2–11.
[31]
Michael P O’Brien, Jim Buckley, and Teresa M Shaft. 2004. Expectation-based, inference-based, and bottom-up software comprehension. Journal of Software Maintenance and Evolution: Research and Practice 16, 6 (2004), 427–447.
[32]
Nancy Pennington. 1987. Comprehension strategies in programming. In Empirical Studies of Programmers: Second Workshop, 1987. 100–113.
[33]
Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology 19, 3 (1987), 295–341.
[34]
Rüdiger F Pohl. 2022. Cognitive Illusions. Cognitive Illusions: Intriguing Phenomena in Thinking, Judgment, and Memory (2022), 3.
[35]
Karl Popper. 2005. The logic of scientific discovery. Routledge.
[36]
James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive difficulties faced by novice programmers in automated assessment tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 41–50.
[37]
Michael Raadt, Mark Toleman, and Richard Watson. 2007. Incorporating programming strategies explicitly into curricula. In Proceedings of the Seventh Baltic Sea Conference on Computing Education Research (Koli Calling 2007).
[38]
Nikolaus Ritt. 2005. Analogy and Transfer: Encoding the Problem at the Right Level of Abstraction. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 27.
[39]
Jorma Sajaniemi. 2002. An empirical analysis of roles of variables in novice-level procedural programs. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. IEEE, 37–39.
[40]
Jorma Sajaniemi, Mordechai Ben-Ari, Pauli Byckling, Petri Gerdt, and Yevgeniya Kulikova. 2006. Roles of variables in three programming paradigms. Computer Science Education 16, 4 (2006), 261–279.
[41]
Jorma Sajaniemi and Marja Kuittinen. 2003. Program animation based on the roles of variables. In Proceedings of the 2003 ACM symposium on Software visualization. 7–ff.
[42]
Jorma Sajaniemi and Marja Kuittinen. 2005. An experiment on using roles of variables in teaching introductory programming. Computer Science Education 15, 1 (2005), 59–82.
[43]
Emmanuel Sander and Jean-François Richard. 1997. Analogical transfer as guided by an abstraction process: The case of learning by doing in text editing.Journal of experimental psychology: learning, memory, and cognition 23, 6 (1997), 1459.
[44]
Carsten Schulte. 2008. Block Model: an educational model of program comprehension as a tool for a scholarly approach to teaching. In Proceedings of the Fourth international Workshop on Computing Education Research. 149–160.
[45]
Daniel L Schwartz. 1993. The construction and analogical transfer of symbolic visualizations. Journal of research in science teaching 30, 10 (1993), 1309–1325.
[46]
Nianfeng Shi. 2021. Improving undergraduate novice programmer comprehension through case-based teaching with roles of variables to provide scaffolding. Information 12, 10 (2021), 424.
[47]
Shuhaida Shuhidan, Margaret Hamilton, and Daryl D’souza. 2009. A taxonomic study of novice programming summative assessment. In Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95. Citeseer, 147–156.
[48]
Elliot Soloway. 1986. Learning to program= learning to construct mechanisms and explanations. Commun. ACM 29, 9 (1986), 850–858.
[49]
Elliot Soloway, Beth Adelson, and Kate Ehrlich. 1988. Knowledge and processes in the comprehension of computer programs. The nature of expertise (1988), 129–152.
[50]
Juha Sorva 2012. Visual program simulation in introductory programming education. Aalto University.
[51]
Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A review of generic program visualization systems for introductory programming education. ACM Transactions on Computing Education (TOCE) 13, 4 (2013), 1–64.
[52]
James C Spohrer and Elliot Soloway. 1986. Novice mistakes: Are the folk wisdoms correct?Commun. ACM 29, 7 (1986), 624–632.
[53]
James C Spohrer, Elliot Soloway, and Edgar Pope. 1985. A goal/plan analysis of buggy Pascal programs. Human–Computer Interaction 1, 2 (1985), 163–207.
[54]
Donna Teague. 2015. Neo-Piagetian theory and the novice programmer. Diss. Queensland University of Technology (2015).
[55]
Donna Teague, Malcolm Corney, Alireza Ahadi, and Raymond Lister. 2013. A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers. In Proceedings of the 15th Australasian Computing Education Conference [Conferences in Research and Practice in Information Technology, Volume 136]. Australian Computer Society, 87–95.
[56]
Donna Teague and Raymond Lister. 2014. Blinded by their Plight: Tracing and the Preoperational Programmer. In PPIG. 8.
[57]
Donna Teague and Raymond Lister. 2014. Longitudinal think aloud study of a novice programmer. In Conferences in Research and Practice in Information Technology Series.
[58]
Vesa Vainio and Jorma Sajaniemi. 2007. Factors in novice programmers’ poor tracing skills. ACM SIGCSE Bulletin 39, 3 (2007), 236–240.
[59]
Iris Vessey. 1985. Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies 23, 5 (1985), 459–494.
[60]
Anneliese Von Mayrhauser and A Marie Vans. 1995. Program comprehension during software maintenance and evolution. Computer 28, 8 (1995), 44–55.
[61]
Renske Weeda, Sjaak Smetsers, and Erik Barendsen. 2023. Unraveling novices’ code composition difficulties. Computer Science Education (2023), 1–28.
[62]
Mark Weiser. 1984. Program Slicing. IEEE Transactions on Software Engineering SE-10, 4 (1984), 352–357. https://doi.org/10.1109/TSE.1984.5010248
[63]
JL Whalley, Tony Clear, Phil Robbins, and Errol Thompson. 2011. Salient elements in novice solutions to code writing problems. (2011).
[64]
Susan Wiedenbeck. 1991. The initial stage of program comprehension. International Journal of Man-Machine Studies 35, 4 (1991), 517–540.
[65]
Susan Wiedenbeck and Jean Scholtz. 1989. Beacons: A knowledge structure in program comprehension. In Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.). 82–87.
[66]
Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Andrew J Ko. 2019. A theory of instruction for introductory programming skills. Computer Science Education 29, 2-3 (2019), 205–253.

Cited By

View all
  • (2024)Evaluating How Novices Utilize Debuggers and Code Execution to Understand CodeProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671126(65-83)Online publication date: 12-Aug-2024

Index Terms

  1. Evaluating Beacons, the Role of Variables, Tracing, and Abstract Tracing for Teaching Novices to Understand Program Intent

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICER '23: Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1
    August 2023
    520 pages
    ISBN:9781450399760
    DOI:10.1145/3568813
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 September 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. CS 1
    2. program comprehension
    3. tracing

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    Conference

    ICER 2023

    Acceptance Rates

    Overall Acceptance Rate 189 of 803 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)152
    • Downloads (Last 6 weeks)22
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Evaluating How Novices Utilize Debuggers and Code Execution to Understand CodeProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671126(65-83)Online publication date: 12-Aug-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media