skip to main content
10.1145/3328778.3366882acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article
Open access

The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior

Published: 26 February 2020 Publication History

Abstract

It is generally accepted that learning to program could be easier for many students. One of the most important components of this experience is the programming environment. Novices learn in a variety of environments, from basic command-line interfaces to industry-strength IDEs. These environments can differ substantially in compilation behavior and error message presentation - arguably two of the most important mechanisms through which users interact with the programming language. In this study, we utilize Blackbox data to compare the programming behavior of thousands of users programming in Java, who all used BlueJ versions 3 and 4. These two versions differ drastically in terms of compilation behavior and error message presentation. BlueJ 3 is a click-to-compile editor that delivers text-based error messages from javac to the user, but only presents the first error message, even if the compiler produces several. BlueJ 4 automatically compiles in the background but retains click-to-compile ability. In addition, all error messages (not just the first) may be viewed by the user. We find that the programming experience and behavior of these users can be substantially affected by changes in these mechanisms, causing numbers of manual compilations, successful compilations, and error messages presented in each version to differ, in cases, markedly. Our results provide evidence on how changes in programming environment affect user behavior in conditions that reasonably control for variables other than the programming environment. This can inform the decisions of educators, tool designers, and HCI researchers in their work to make learning more effective for novice programmers.

References

[1]
Eric Allen, Robert Cartwright, and Brian Stoler. 2002. DrJava: A Lightweight Pedagogic Environment for Java. In Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Education (SIGCSE '02). ACM, New York, NY, USA, 137--141. https://doi.org/10.1145/563340.563395
[2]
American Psychological Association. 2009. Publication Manual 6th ed.). American Psychological Association, Washington, DC.
[3]
Titus Barik. 2018. Error Messages as Rational Reconstructions . Ph.D. Dissertation. North Carolina State University, Raleigh, NC, USA.
[4]
Brett A. Becker. 2016. An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE '16). ACM, New York, NY, USA, 126--131. https://doi.org/10.1145/2839509.2844584
[5]
Brett A. Becker and Thomas Fitzpatrick. 2019. What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 1011--1017. https://doi.org/10.1145/3287324.3287485
[6]
Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018a. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 640--645. https://doi.org/10.1145/3159450.3159461
[7]
Brett A. Becker and Catherine Mooney. 2016. Categorizing compiler error messages with principal component analysis. In 12th China-Europe International Symposium on Software Engineering Education (CEISEE 2016), Shenyang, China, 28--29 May 2016 .
[8]
Brett A. Becker, Cormac Murray, Tianyi Tao, Changheng Song, Robert McCartney, and Kate Sanders. 2018b. Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 634--639. https://doi.org/10.1145/3159450.3159453
[9]
Dennis Bouvier, Ellie Lovellette, John Matta, Bedour Alshaigy, Brett A. Becker, Michelle Craig, Jana Jackova, Robert McCartney, Kate Sanders, and Mark Zarb. 2016. Novice Programmers and the Problem Description Effect. In Proceedings of the 2016 ITiCSE Working Group Reports (ITiCSE '16). ACM, New York, NY, USA, 103--118. https://doi.org/10.1145/3024906.3024912
[10]
Neil C. C. Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018. Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). ACM, New York, NY, USA, 196--204. https://doi.org/10.1145/3230977.3230991
[11]
Neil C. C. Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers' Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). ACM, New York, NY, USA, 223--228. https://doi.org/10.1145/2538862.2538924
[12]
Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences .Routledge. https://doi.org/10.4324/9780203771587
[13]
Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing Syntax Error Messages Appears Ineffectual. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (ITiCSE '14). ACM, New York, NY, USA, 273--278. https://doi.org/10.1145/2591708.2591748
[14]
Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All Syntax Errors Are Not Equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE '12). ACM, New York, NY, USA, 75--80. https://doi.org/10.1145/2325296.2325318
[15]
Edward Dillon, Monica Anderson, and Marcus Brown. 2012. Comparing Feature Assistance Between Programming Environments and Their "Effect" on Novice Programmers. J. Comput. Sci. Coll., Vol. 27, 5 (May 2012), 69--77. http://dl.acm.org/citation.cfm?id=2168874.2168894
[16]
Tao Dong and Kandarp Khandwala. 2019. The Impact of "Cosmetic" Changes on the Usability of Error Messages. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Article LBW0273, bibinfonumpages6 pages. https://doi.org/10.1145/3290607.3312978
[17]
Christoph Hannebauer, Marc Hesenius, and Volker Gruhn. 2018. Does syntax highlighting help programming novices? Empirical Software Engineering, Vol. 23, 5 (01 Oct 2018), 2795--2828. https://doi.org/10.1007/s10664-017--9579-0
[18]
Matthew Hertz. 2010. What Do "CS1" and "CS2" Mean?: Investigating Differences in the Early Courses. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (SIGCSE '10). ACM, New York, NY, USA, 199--203. https://doi.org/10.1145/1734263.1734335
[19]
Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE on Working Group Reports (ITICSE-WGR '15). ACM, New York, NY, USA, 41--63. https://doi.org/10.1145/2858796.2858798
[20]
Jam Jenkins, Evelyn Brannock, Thomas Cooper, Sonal Dekhane, Mark Hall, and Michael Nguyen. 2012. Perspectives on Active Learning and Collaboration: JavaWIDE in the Classroom. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (SIGCSE '12). ACM, New York, NY, USA, 185--190. https://doi.org/10.1145/2157136.2157194
[21]
Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg. 2007. A systematic review of effect size in software engineering experiments. Information and Software Technology, Vol. 49, 11--12 (2007), 1073--1086.
[22]
Ioannis Karvelas. 2019. Investigating Novice Programmers' Interaction with Programming Environments. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 336--337. https://doi.org/10.1145/3304221.3325596
[23]
Amy J. Ko and Brad A. Myers. 2008. Debugging Reinvented: Asking and Answering Why and Why Not Questions About Program Behavior. In Proceedings of the 30th International Conference on Software Engineering (ICSE '08). ACM, New York, NY, USA, 301--310. https://doi.org/10.1145/1368088.1368130
[24]
Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ System and its Pedagogy. Computer Science Education, Vol. 13, 4 (2003), 249--268. https://doi.org/10.1076/csed.13.4.249.17496
[25]
Wanda M. Kunkle and Robert B. Allen. 2016. The impact of different teaching approaches and languages on student learning of introductory programming concepts. ACM Transactions on Computing Education (TOCE), Vol. 16, 1 (2016), 3:1--3:26.
[26]
Johnson Ching-Hong Li. 2016. Effect size measures in a two-independent-samples case with nonnormal and nonhomogeneous data. Behavior Research Methods, Vol. 48, 4 (01 Dec 2016), 1560--1574. https://doi.org/10.3758/s13428-015-0667-z
[27]
Andrew Luxton-Reilly. 2016. Learning to Program is Easy. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 284--289. https://doi.org/10.1145/2899415.2899432
[28]
Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE 2018 Companion). ACM, New York, NY, USA, 55--106. https://doi.org/10.1145/3293881.3295779
[29]
H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, Vol. 18, 1 (1947), 50--60. http://www.jstor.org/stable/2236101
[30]
Geoff Norman. 2010. Likert scales, levels of measurement and the laws of statistics. Advances in health sciences education, Vol. 15, 5 (2010), 625--632.
[31]
James Prather, Raymond Pettit, Brett A. Becker, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht, and Krista Masci. 2019. First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 531--537. https://doi.org/10.1145/3287324.3287374
[32]
James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017. On Novices' Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research (ICER '17). ACM, New York, NY, USA, 74--82. https://doi.org/10.1145/3105726.3106169
[33]
Charles Reis and Robert Cartwright. 2004. Taming a Professional IDE for the Classroom. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '04). ACM, New York, NY, USA, 156--160. https://doi.org/10.1145/971300.971357
[34]
Peter C. Rigby and Suzanne Thompson. 2005. Study of Novice Programmers Using Eclipse and Gild. In Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology eXchange (eclipse '05). ACM, New York, NY, USA, 105--109. https://doi.org/10.1145/1117696.1117718
[35]
Kate Sanders, Judy Sheard, Brett A. Becker, Anna Eckerdal, Sally Hamouda, and Simon. 2019. Inferential Statistics in Computing Education Research: A Methodological Review. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 177--185. https://doi.org/10.1145/3291279.3339408
[36]
Mark Santolucito, William T. Hallahan, and Ruzica Piskac. 2019. Live Programming By Example. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, New York, NY, USA, Article INT020, bibinfonumpages4 pages. https://doi.org/10.1145/3290607.3313266
[37]
Advait Sarkar. 2015. The Impact of Syntax Colouring on Program Comprehension. In Proceedings of the 26th Psychology of Programming Interest Group . http://www.ppig.org/library/paper/impact-syntax-colouring-program-comprehension
[38]
S. S. Shapiro and M. B. Wilk. 1965. An Analysis of Variance Test for Normality (Complete Samples). Biometrika, Vol. 52, 3/4 (1965), 591--611. http://www.jstor.org/stable/2333709
[39]
M. E. Sime, A. T. Arblaster, and T. R. Green. 1977. Structuring the Programmer's Task . Journal of Occupational Psychology, Vol. 50, 3 (sep 1977), 205--216. https://doi.org/10.1111/j.2044--8325.1977.tb00376.x
[40]
Andreas Stefik and Susanna Siebert. 2013. An Empirical Investigation into Programming Language Syntax. Trans. Comput. Educ., Vol. 13, 4, Article 19 (Nov. 2013), bibinfonumpages40 pages. https://doi.org/10.1145/2534973

Cited By

View all
  • (2024)DCC Sidekick: Helping Novices Solve Programming Errors Through a Conversational Explanation InterfaceProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635483(1714-1715)Online publication date: 14-Mar-2024
  • (2024)dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language ModelsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630822(1314-1320)Online publication date: 7-Mar-2024
  • (2024)LLM-aided explanations of EDA synthesis errors2024 IEEE LLM Aided Design Workshop (LAD)10.1109/LAD62341.2024.10691721(1-6)Online publication date: 28-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education
February 2020
1502 pages
ISBN:9781450367936
DOI:10.1145/3328778
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 February 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. blackbox
  2. bluej
  3. compiler error messages
  4. cs1
  5. editors
  6. educational data mining
  7. ide
  8. novice programmers
  9. programming
  10. programming environments
  11. programming process data
  12. tools

Qualifiers

  • Research-article

Conference

SIGCSE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025
The 56th ACM Technical Symposium on Computer Science Education
February 26 - March 1, 2025
Pittsburgh , PA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)163
  • Downloads (Last 6 weeks)19
Reflects downloads up to 26 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)DCC Sidekick: Helping Novices Solve Programming Errors Through a Conversational Explanation InterfaceProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635483(1714-1715)Online publication date: 14-Mar-2024
  • (2024)dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language ModelsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630822(1314-1320)Online publication date: 7-Mar-2024
  • (2024)LLM-aided explanations of EDA synthesis errors2024 IEEE LLM Aided Design Workshop (LAD)10.1109/LAD62341.2024.10691721(1-6)Online publication date: 28-Jun-2024
  • (2023)“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice ProgrammersACM Transactions on Computer-Human Interaction10.1145/361736731:1(1-31)Online publication date: 29-Nov-2023
  • (2023)Programmers' Views on IDE Compilation MechanismsProceedings of the ACM Conference on Global Computing Education Vol 110.1145/3576882.3617915(98-104)Online publication date: 5-Dec-2023
  • (2023)My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming ExercisesProceedings of the 25th Australasian Computing Education Conference10.1145/3576123.3576134(97-104)Online publication date: 30-Jan-2023
  • (2023)Foundations FirstProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569768(346-352)Online publication date: 2-Mar-2023
  • (2023)Programming Is Hard - Or at Least It Used to BeProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569759(500-506)Online publication date: 2-Mar-2023
  • (2023)Analyse and Evaluate Quixbugs with Open AI Codex and Powering Next Generation Application2023 6th International Conference on Contemporary Computing and Informatics (IC3I)10.1109/IC3I59117.2023.10397753(165-170)Online publication date: 14-Sep-2023
  • (2023)IDE Interactions of Novices Transitioning Between Programming EnvironmentsTowards a Collaborative Society Through Creative Learning10.1007/978-3-031-43393-1_54(603-614)Online publication date: 28-Sep-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media