skip to main content
10.1145/1953163.1953308acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article

Measuring the effectiveness of error messages designed for novice programmers

Published: 09 March 2011 Publication History

Abstract

Good error messages are critical for novice programmers. Re-cognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a course-worth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages.

References

[1]
Chabert, J.M. and Higginbotham, T.F. An Investigation of Novice Programmer Errors in IBM 370 (OS) Assembly Lan-guage. In Proceedings of the ACM Southeast Regional Confe-rence, pages 319--323. 1976.
[2]
Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37--46, 1960.
[3]
Coull, N.J. SNOOPIE: Development Of A Learning Support Tool For Novice Programmers Within A Conceptual Framework. PhD Thesis, School of Computer Science, University Of St. Andrews, 2008.
[4]
Fagin, B.S. and Merkle, L. Quantitative analysis of the effects of robots on introductory Computer Science education. Journal on Educational Resources in Computing, 2(4):1--18, 2002.
[5]
Findler, R.B., Clements, J., Flanagan, C., Flatt, M., Krishna-murthi, S., Steckler, P., and Felleisen, M. DrScheme: A programming environment for Scheme. Journal of Functional Programming, 12(02):159--182, 2002.
[6]
Gross, P. and Powers, K. Evaluating assessments of novice programming environments. Proceedings of the International Workshop on Computing Education Research, :99--110, 2005.
[7]
Hagan, D. and Markham, S. Teaching Java with the BlueJ environment. In Proceedings of Australasian Society for Computers in Learning in Tertiary Education Conference. Citeseer, 2000.
[8]
Hage, J. and Keeken, P.V. Mining Helium programs with Neon. In Technical Report, Department of Information and Computing Sciences, Utrecht University.
[9]
Holt, R.C., Wortman, D.B., Barnard, D.T., and Cordy, J.R. SP/k: a system for teaching computer programming. Communications of the ACM, 20(5):301--309, 1977.
[10]
Hristova, M., Misra, A., Rutter, M., and Mercuri, R. Identifying and correcting Java programming errors for introductory computer science students. In Proceedings of the Symposium on Computer Science Education, pages 153--156. 2003.
[11]
Jackson, J., Cobb, M., and Carver, C. Identifying top Java errors for novice programmers. In Proceedings of the Frontiers in Education Conference, pages T4C--24. 2005.
[12]
Jadud, M.C. A First Look at Novice Compilation Behaviour Using BlueJ. Computer Science Education, 15(1):25--40, 2005.
[13]
Jadud, M.C. Methods and tools for exploring novice compilation behaviour. In Proceedings of the International Work-shop on Computing Education Research, pages 73--84. ACM, 2006.
[14]
Köksal, M.F., Baar, R.E., and Üsküdarl1, S. Screen-Replay: A Session Recording and Analysis Tool for DrScheme. In Proceedings of the Scheme and Functional Programming Workshop, Technical Report, California Polytechnic State University, CPSLO-CSC-09-03, pages 103--110. 2009.
[15]
Lane, H.C. and VanLehn, K. Intention-based scoring: An approach to measuring success at solving the composition problem. ACM SIGCSE Bulletin, 37(1):373--377, 2005.
[16]
Levy, R.B., Ben-Ari, M., and Uronen, P.A. The Jeliot 2000 program animation system. Computers & Education, 40(1):1--15, 2003.
[17]
Moskal, B., Lurie, D., and Cooper, S. Evaluating the effectiveness of a new instructional approach. Proceedings of the Symposium on Computer Science Education, 35:75--79, 2004.
[18]
Murphy, L., Lewandowski, G., McCauley, R., Simon, B., Thomas, L., and Zander, C. Debugging: the good, the bad, and the quirky - a qualitative analysis of novices' strategies. ACM SIGCSE Bulletin, 40(1):163--167, 2008.
[19]
Ng Cheong Vee, M., Mannock, K., and Meyer, B. Empirical study of novice errors and error paths in object-oriented programming. In Proceedings of the Conference of the Higher Education Academy, Subject Centre for Information and Computer Sciences, pages 54--58. 2006.
[20]
Nienaltowski, M., Pedroni, M., and Meyer, B. Compiler Error Messages: What Can Help Novices? Proceedings of the Technical Symposium on Computer Science Education, 39:168--172, 2008.
[21]
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., and Paterson, J. A survey of literature on the teaching of introductory programming. ACM SIGCSE Bulletin, 39(4):204--223, 2007.
[22]
Ragonis, N. and Ben-Ari, M. On understanding the statics and dynamics of object-oriented programs. ACM SIGCSE Bulletin, 37(1):226--230, 2005.
[23]
Rey, J.S. From Alice To BlueJ: A Transition To Java. Master's thesis, School of Computing, Robert Gordon University, 2009.
[24]
Spohrer, J.C. and Soloway, E. Novice mistakes: are the folk wisdoms correct? Communications of the ACM, 29(7)1986.
[25]
Van Haaster, K. and Hagan, D. Teaching and Learning with BlueJ: an Evaluation of a Pedagogical Tool. In Information Science + Information Technology Education Joint Conference. Citeseer, 2004.

Cited By

View all
  • (2024)Non-Expert Programmers in the Generative AI FutureProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663393(1-19)Online publication date: 25-Jun-2024
  • (2024)Feedback Literacy: Holistic Analysis of Secondary Educators' Views of LLM Explanations of Program Error MessagesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653595(192-198)Online publication date: 3-Jul-2024
  • (2024)"In the Beginning, I Couldn't Necessarily Do Anything With It": Links Between Compiler Error Messages and Sense of BelongingProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671105(14-26)Online publication date: 12-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGCSE '11: Proceedings of the 42nd ACM technical symposium on Computer science education
March 2011
754 pages
ISBN:9781450305006
DOI:10.1145/1953163
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. error messages
  2. novice programmers
  3. user-studies

Qualifiers

  • Research-article

Conference

SIGCSE '11
Sponsor:

Acceptance Rates

SIGCSE '11 Paper Acceptance Rate 107 of 315 submissions, 34%;
Overall Acceptance Rate 1,787 of 5,146 submissions, 35%

Upcoming Conference

SIGCSE TS 2025
The 56th ACM Technical Symposium on Computer Science Education
February 26 - March 1, 2025
Pittsburgh , PA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)62
  • Downloads (Last 6 weeks)10
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Non-Expert Programmers in the Generative AI FutureProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663393(1-19)Online publication date: 25-Jun-2024
  • (2024)Feedback Literacy: Holistic Analysis of Secondary Educators' Views of LLM Explanations of Program Error MessagesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653595(192-198)Online publication date: 3-Jul-2024
  • (2024)"In the Beginning, I Couldn't Necessarily Do Anything With It": Links Between Compiler Error Messages and Sense of BelongingProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671105(14-26)Online publication date: 12-Aug-2024
  • (2024)On Students’ Perception of Compiler Syntax Error Messages: A Human Factors ApproachHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76815-6_1(3-16)Online publication date: 11-Dec-2024
  • (2024)Error Messages for Students Taught Using a Systematic Program Design CurriculumTrends in Functional Programming10.1007/978-3-031-74558-4_9(195-219)Online publication date: 8-Jan-2024
  • (2023)Regular Expressions in a CS Formal Languages CourseElectronic Proceedings in Theoretical Computer Science10.4204/EPTCS.382.3382(45-63)Online publication date: 14-Aug-2023
  • (2023)How Domain Experts Use an Embedded DSLProceedings of the ACM on Programming Languages10.1145/36228517:OOPSLA2(1499-1530)Online publication date: 16-Oct-2023
  • (2023)A Grounded Conceptual Model for Ownership Types in RustProceedings of the ACM on Programming Languages10.1145/36228417:OOPSLA2(1224-1252)Online publication date: 16-Oct-2023
  • (2023)Getting into the Flow: Towards Better Type Error Messages for Constraint-Based Type InferenceProceedings of the ACM on Programming Languages10.1145/36228127:OOPSLA2(431-459)Online publication date: 16-Oct-2023
  • (2023)Mind the Error Message: An Inverted Quiz Format to Direct Learner's Attention to Error MessagesProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588823(382-388)Online publication date: 29-Jun-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media