skip to main content
10.1145/3649217.3653595acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

Feedback Literacy: Holistic Analysis of Secondary Educators' Views of LLM Explanations of Program Error Messages

Published: 03 July 2024 Publication History

Abstract

The implications of using large language model (LLM) tools for learning to program at secondary school level are largely unknown, and yet there is pressure for teachers to engage with these. To start addressing this gap, we investigated: RQ1: What are secondary educators' views on the potential classroom use of LLM program error message explanations? RQ2: In what ways can a feedback literacy perspective support the analysis of educators' views of potential classroom use of LLM program error message explanations? The responses of eight expert secondary school educators were gathered during a semi-structured, activity-based interview and qualitatively analysed. Fifteen themes were derived from their commentary, of which ten corresponded to enhanced program error message (PEM) guidelines. Yet, all themes correlated to feedback literacy theory, providing a more holistic view. The analysis revealed that educators preferred LLM explanations to guide and develop understanding rather than tell, that students should be supported to make judgements and action LLM-generated feedback. Combining PEM guideline and feedback literacy findings, we suggest augmented IDEs should be designed with educators and students in mind, and teacher professional development (PD) is needed. Research is needed to compare our findings with a wider range of educators and investigate what feedback literacy means for resource design, PD, and classroom practice in secondary and undergraduate contexts.

References

[1]
Brett Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice Pearce, and James Prather. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (Aberdeen, Scotland Uk) (ITiCSE-WGR '19). ACM, New York, NY, USA, 177--210. https://doi.org/10.1145/3344429.3372508
[2]
Brett Becker, Graham Glanville, Ricardo Iwashima, Claire Mcdonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective compiler error message enhancement for novice programming students. Computer Science Education, Vol. 26, 2--3 (09 2016), 148--175. https://doi.org/10.1080/08993408.2016.1225464
[3]
Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 500--506. https://doi.org/10.1145/3545945.3569759
[4]
Paul Black and Dylan Wiliam. 2010. Inside the Black Box Raising Standards Through Classroom Assessment., Vol. 80 (09 2010). https://doi.org/10.1177/003172171009200119
[5]
David Boud and Elizabeth Molloy. 2013. Rethinking models of feedback for learning. Assessment & Evaluation in higher education, Vol. 38, 6 (2013), 698--712.
[6]
British Educational Research Association. 2018. Ethical guidelines for educational research 4 ed.). British Educational Research Association, London. https://www.bera.ac.uk/resources/all-publications/resources-for-researchers
[7]
Cambridge University Press. [n.,d.]. LLM. Cambridge Dictionary. Accessed 26-03--2024. https://dictionary.cambridge.org/dictionary/english/large-language-model
[8]
David Carless and David Boud. 2018. The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education, Vol. 43, 8 (2018), 1315--1325. https://doi.org/10.1080/02602938.2018.1463354
[9]
David Carless and Naomi Winstone. 2023. Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education, Vol. 28, 1 (2023), 150--163.
[10]
Paul Denny, James Prather, and Brett A. Becker. 2020. Error Message Readability and Novice Debugging Performance. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE '20). ACM, New York, NY, USA, 480--486. https://doi.org/10.1145/3341525.3387384
[11]
Abigail Evans, Zihan Wang, Jieren Liu, and Mingming Zheng. 2023. SIDE-Lib: A Library for Detecting Symptoms of Python Programming Misconceptions. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 159--165. https://doi.org/10.1145/3587102.3588838
[12]
John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, Vol. 77, 1 (2007), 81--112.
[13]
Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of LLMs to Beginner Programmers' Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (Chicago, IL, USA) (ICER '23). ACM, New York, NY, USA, 93--105. https://doi.org/10.1145/3568813.3600139
[14]
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, and et al. 2023. ChatGPT for Good? On Opportunities and Challenges of LLMs for Education. Learning and Individual Differences, Vol. 103, 102274 (Jan 2023). https://doi.org/10.35542/osf.io/5er8f
[15]
Majeed Kazemitabaar, Justin Chow, Carl Ma, Barbara Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). ACM, New York, NY, USA, Article 455, 23 pages. https://doi.org/10.1145/3544548.3580919
[16]
Udo Kuckartz. 2022. Qualitative Text Analysis: A Guide to Methods, Practice & Using Software. Sage Publications, London. https://doi.org/10.4135/9781446288719
[17]
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, Vol. 33, 1 (1977), 159--174.
[18]
Sam Lau and Philip Guo. 2023. From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER '23). ACM, New York, NY, USA, 106--121. https://doi.org/10.1145/3568813.3600138
[19]
Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and LLMs. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 124--130. https://doi.org/10.1145/3587102.3588785
[20]
Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2022. Using LLMs to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 563--569. https://doi.org/10.1145/3545945.3569770
[21]
Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating Diverse Code Explanations Using the GPT-3 LLM. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 (Lugano and Virtual Event, Switzerland) (ICER '22). ACM, New York, NY, USA, 37--39. https://doi.org/10.1145/3501709.3544280
[22]
Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Measuring the Effectiveness of Error Messages Designed for Novice Programmers. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE '11). ACM, New York, NY, USA, 499--504. https://doi.org/10.1145/1953163.1953308
[23]
Philipp Mayring. 2000. Qualitative Content Analysis. In Forum Qualitative Sozialforschung / Forum: Qualitative Social Research,. Vol. 1 (2). Online. Issue Art.20. https://doi.org/10.17169/fqs-1.2.1089 Accessed 14--11--2022.
[24]
Davin McCall and Michael Kölling. 2019. A New Look at Novice Programmer Errors. ACM Trans. Comput. Educ., Vol. 19, 4, Article 38 (jul 2019), 30 pages. https://doi.org/10.1145/3335814
[25]
Angela J McLean, Carol H Bond, and Helen D Nicholson. 2015. An anatomy of feedback: a phenomenographic investigation of undergraduate students' conceptions of feedback. Studies in Higher Education, Vol. 40, 5 (2015), 921--932. https://doi.org/10.1080/03075079.2013.855718
[26]
Elizabeth Molloy, David Boud, and Michael Henderson. 2020. Developing a learning-centred framework for feedback literacy. Assessment & Evaluation in Higher Education, Vol. 45, 4 (2020), 527--540.
[27]
OCR. 2023. GCSE Computer Science: What programming language should I use for GCSE. Online. https://support.ocr.org.uk/hc/en-gb/articles/10066252791698-GCSE-Computer-Science-What-programming-language-should-I-use-for-GCSE- Accessed 19--12--2023.
[28]
OpenAI. 2023. GPT-3.5. https://platform.openai.com/docs/models/gpt-3--5
[29]
OpenAI. 2023. How ChatGPT and Our Language Models Are Developed. https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-language-models-are-developed
[30]
James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (Turku,Finland) (ITiCSE-WGR '23). ACM, New York, NY, USA, 108--159. https://doi.org/10.1145/3623762.3633499
[31]
Raspberry Pi Foundation. 2023. Using large language models to explain programming error messages research study website. http://rpf.io/llm-pem-pilot
[32]
Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and teaching programming: A review and discussion. Computer science education, Vol. 13, 2 (2003), 137--172.
[33]
Tracii Ryan, Michael Henderson, Kris Ryan, and Gregor Kennedy. 2023. Identifying the components of effective learner-centred feedback information. Teaching in Higher Education, Vol. 28, 7 (2023), 1565--1582.
[34]
Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using LLMs. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22). ACM, New York, NY, USA, 27--43. https://doi.org/10.1145/3501385.3543957
[35]
Cory Stig. 2023. Can Generative AI Teach You to Code? It's Complicated. https://www.codecademy.com/resources/blog/can-chatgpt-ai-teach-you-to-code/
[36]
Paul Sutton. 2012. Conceptualizing feedback literacy: knowing, being, and acting. Innovations in Education and Teaching International, Vol. 49 (02 2012), 31--40. https://doi.org/10.1080/14703297.2012.647781
[37]
The Chartered Institute for IT. 2023. Will AI replace software engineers? https://www.bcs.org/articles-opinion-and-research/will-ai-replace-software-engineers/
[38]
Warren Toomey. 2011. Quantifying the incidence of novice programmers' errors. School of IT, Bond University (2011). https://api.semanticscholar.org/CorpusID:14453637
[39]
V. Javier Traver. 2010. On Compiler Error Messages: What They Say and What They Mean. Adv. in Hum.-Comp. Int., Vol. 2010, Article 3 (Jan 2010), 26 pages. https://doi.org/10.1155/2010/602570
[40]
Jane Waite, Eirini Kolaiti, Meurig Thomas, and Karl Maton. 2023. Constructing feedback for computer science MCQ wrong answers using semantic profiling. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research. ACM, New York, NY, USA.
[41]
C. Zastudil, M. Rogalska, C. Kapp, J. Vaughn, and S. MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. In 2023 IEEE Frontiers in Education Conference (FIE). IEEE Computer Society, Los Alamitos, CA, USA, 1--9. https://doi.org/10.1109/FIE58773.2023.10343467

Cited By

View all
  • (2024)Some theories from abroad for AI interaction literacy.Proceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689563(1-3)Online publication date: 5-Sep-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1
July 2024
776 pages
ISBN:9798400706004
DOI:10.1145/3649217
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 July 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. AI
  2. IDE
  3. K-12 education
  4. ML
  5. feedback literacy

Qualifiers

  • Research-article

Conference

ITiCSE 2024
Sponsor:

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Upcoming Conference

ITiCSE '25
Innovation and Technology in Computer Science Education
June 27 - July 2, 2025
Nijmegen , Netherlands

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)221
  • Downloads (Last 6 weeks)27
Reflects downloads up to 11 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Some theories from abroad for AI interaction literacy.Proceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689563(1-3)Online publication date: 5-Sep-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media