research-article

Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback

Authors:

Jacqueline Whalley,

Juho LeinonenAuthors Info & Claims

ACE '21: Proceedings of the 23rd Australasian Computing Education Conference

Pages 88 - 95

https://doi.org/10.1145/3441636.3442309

Published: 17 March 2021 Publication History

Abstract

Programming assignments are a common form of assessment in introductory courses and often require substantial work to complete. Students must therefore plan and manage their time carefully, especially leading up to published deadlines. Although time management is an important metacognitive skill that students must develop, it is rarely taught explicitly. Prior research has explored various approaches for reducing procrastination and other unproductive behaviours in students, but these are often ineffective or impractical in large courses. In this work, we investigate a scalable intervention that incentivizes students to begin work early. We provide automatically generated feedback to students who submit their work-in-progress prior to two fixed deadlines scheduled earlier than the final deadline for the assignment. Although voluntary, we find that many students welcome this early feedback and improve the quality of their work across each iteration. Especially for at-risk students, who have failed an earlier module in the course, engaging with the early feedback opportunities results in significantly better work at the time of final submission.

References

[1]

Tapio Auvinen. 2015. Harmful Study Habits in Online Learning Environments with Automatic Assessment. In 2015 International Conference on Learning and Teaching in Computing and Engineering. 50–57.

Digital Library

[2]

Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research. In Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education(ITiCSE-WGR ’19). ACM, New York, NY, USA, 177–210. https://doi.org/10.1145/3344429.3372508

Digital Library

[3]

Brent J. Bowman and William A. Newman. 1990. Software Metrics as a Programming Training Tool. J. Syst. Softw. 13, 2 (Oct. 1990), 139–147. https://doi.org/10.1016/0164-1212(90)90119-7

Digital Library

[4]

Rachel Cardell-Oliver. 2011. How Can Software Metrics Help Novice Programmers?. In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114(ACE ’11). Australian Computer Society, Inc., AUS, 55–62.

[5]

Rachel Cardell-Oliver, Lu Zhang, Rieky Barady, You Hai Lim, Asad Naveed, and Terry Woodings. 2010. Automated Feedback for Quality Assurance in Software Engineering Education. In Proceedings of the 2010 21st Australian Software Engineering Conference(ASWEC ’10). IEEE Computer Society, USA, 157–164. https://doi.org/10.1109/ASWEC.2010.24

Digital Library

[6]

Rohan Roy Choudhury, HeZheng Yin, Joseph Moghadam, and Armando Fox. 2016. AutoStyle: Toward Coding Style Feedback At Scale. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion(CSCW ’16 Companion). Association for Computing Machinery, New York, NY, USA, 21–24. https://doi.org/10.1145/2818052.2874315

Digital Library

[7]

Sophie H. Cormack, Laurence A. Eagle, and Mark S. Davies. 2020. A large-scale test of the relationship between procrastination and performance using learning analytics. Assessment & Evaluation in Higher Education 0, 0 (2020), 1–14. https://doi.org/10.1080/02602938.2019.1705244 arXiv:https://doi.org/10.1080/02602938.2019.1705244

[8]

Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Banaszkiewicz. 2019. Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer. In Proceedings of the 2019 ACM Conference on International Computing Education Research(ICER ’19). Association for Computing Machinery, New York, NY, USA, 259–267. https://doi.org/10.1145/3291279.3339402

Digital Library

[9]

Paul Denny, Andrew Luxton-Reilly, Michelle Craig, and Andrew Petersen. 2018. Improving Complex Task Performance Using a Sequence of Simple Practice Tasks. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE 2018). Association for Computing Machinery, New York, NY, USA, 4–9. https://doi.org/10.1145/3197091.3197141

Digital Library

[10]

Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-Driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education(SIGCSE ’11). Association for Computing Machinery, New York, NY, USA, 471–476. https://doi.org/10.1145/1953163.1953299

Digital Library

[11]

Paul Denny, James Prather, and Brett A. Becker. 2020. Error Message Readability and Novice Debugging Performance. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 480–486. https://doi.org/10.1145/3341525.3387384

Digital Library

[12]

Stephen H. Edwards, Joshua Martin, and Clfford A. Shaffer. 2015. Examining Classroom Interventions to Reduce Procrastination. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’15). Association for Computing Machinery, New York, NY, USA, 254–259. https://doi.org/10.1145/2729094.2742632

Digital Library

[13]

Stephen H. Edwards and Krishnan Panamalai Murali. 2017. CodeWorkout: Short Programming Exercises with Built-in Data Collection. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’17). Association for Computing Machinery, New York, NY, USA, 188–193. https://doi.org/10.1145/3059009.3059055

Digital Library

[14]

Stephen H. Edwards and Manuel A. Perez-Quinones. 2008. Web-CAT: Automatically Grading Programming Assignments. In Proceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education(ITiCSE ’08). Association for Computing Machinery, New York, NY, USA, 328. https://doi.org/10.1145/1384271.1384371

Digital Library

[15]

Stephen H. Edwards, Jason Snyder, Manuel A. Pérez-Quiñones, Anthony Allevato, Dongkwan Kim, and Betsy Tretola. 2009. Comparing Effective and Ineffective Behaviors of Student Programmers. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop(ICER ’09). Association for Computing Machinery, New York, NY, USA, 3–14. https://doi.org/10.1145/1584322.1584325

Digital Library

[16]

Tommy Färnqvist and Fredrik Heintz. 2016. Competition and Feedback through Automated Assessment in a Data Structures and Algorithms Course. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’16). Association for Computing Machinery, New York, NY, USA, 130–135. https://doi.org/10.1145/2899415.2899454

Digital Library

[17]

James B. Fenwick, Cindy Norris, Frank E. Barry, Josh Rountree, Cole J. Spicer, and Scott D. Cheek. 2009. Another Look at the Behaviors of Novice Programmers. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education(SIGCSE ’09). Association for Computing Machinery, New York, NY, USA, 296–300. https://doi.org/10.1145/1508865.1508973

Digital Library

[18]

Yoshiko Goda, Masanori Yamada, Hiroshi Kato, Takeshi Matsuda, Yutaka Saito, and Hiroyuki Miyagawa. 2015. Procrastination and other learning behavioral types in e-learning and their relationship with learning outcomes. Learning and Individual Differences 37 (2015), 72 – 80. https://doi.org/10.1016/j.lindif.2014.11.001

[19]

Keith Gregory and Sue Morón-García. 2009. Assignment submission, student behaviour and experience. Engineering Education 4, 1 (2009), 16–28. https://doi.org/10.11120/ened.2009.04010016

[20]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research(Koli Calling ’10). Association for Computing Machinery, New York, NY, USA, 86–93. https://doi.org/10.1145/1930464.1930480

Digital Library

[21]

Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE on Working Group Reports(ITICSE-WGR ’15). Association for Computing Machinery, New York, NY, USA, 41–63. https://doi.org/10.1145/2858796.2858798

Digital Library

[22]

Kalle Ilves, Juho Leinonen, and Arto Hellas. 2018. Supporting self-regulated learning with visualizations in online learning environments. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. 257–262.

Digital Library

[23]

Michael S. Irwin and Stephen H. Edwards. 2019. Can Mobile Gaming Psychology Be Used to Improve Time Management on Programming Assignments?. In Proceedings of the ACM Conference on Global Computing Education(CompEd ’19). Association for Computing Machinery, New York, NY, USA, 208–214. https://doi.org/10.1145/3300115.3309517

Digital Library

[24]

Lucy Jiang, Robert Rewcastle, Paul Denny, and Ewan Tempero. 2020. CompareCFG: Providing Visual Feedback on Code Quality Using Control Flow Graphs. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 493–499. https://doi.org/10.1145/3341525.3387362

Digital Library

[25]

Ayaan M. Kazerouni, Stephen H. Edwards, and Clifford A. Shaffer. 2017. Quantifying Incremental Development Practices and Their Relationship to Procrastination. In Proceedings of the 2017 ACM Conference on International Computing Education Research(ICER ’17). Association for Computing Machinery, New York, NY, USA, 191–199. https://doi.org/10.1145/3105726.3106180

Digital Library

[26]

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. ACM Trans. Comput. Educ. 19, 1, Article Article 3 (Sept. 2018), 43 pages. https://doi.org/10.1145/3231711

Digital Library

[27]

Pardha Koyya, Lee Young, and Jeong Yang. 2013. Feedback for Programming Assignments Using Software-Metrics and Reference Code. ISRN Software Engineering(2013). https://doi.org/10.1155/2013/805963

[28]

Lauri Malmi, Ville Karavirta, Ari Korhonen, and Jussi Nikander. 2005. Experiences on Automatically Assessed Algorithm Simulation Exercises with Different Resubmission Policies. J. Educ. Resour. Comput. 5, 3 (Sept. 2005), 7–es. https://doi.org/10.1145/1163405.1163412

Digital Library

[29]

Keir Mierle, Kevin Laven, Sam Roweis, and Greg Wilson. 2005. Mining Student CVS Repositories for Performance Indicators. In Proceedings of the 2005 International Workshop on Mining Software Repositories(MSR ’05). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/1083142.1083150

Digital Library

[30]

Nick Parlante. 2007. Nifty Reflections. SIGCSE Bull. 39, 2 (June 2007), 25–26. https://doi.org/10.1145/1272848.1272876

Digital Library

[31]

James Prather, Brett A. Becker, Michelle Craig, Paul Denny, Dastyni Loksa, and Lauren Margulieux. 2020. What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming. In Proceedings of the 2020 ACM Conference on International Computing Education Research(ICER ’20). Association for Computing Machinery, New York, NY, USA, 12. https://doi.org/10.1145/3372782.3406263

Digital Library

[32]

Paul Ramsden. 1992. Learning to Teach in Higher Education. Routledge, London.

[33]

Jaime Spacco, Paul Denny, Brad Richards, David Babcock, David Hovemeyer, James Moscola, and Robert Duvall. 2015. Analyzing Student Work Patterns Using Programming Exercise Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education(SIGCSE ’15). Association for Computing Machinery, New York, NY, USA, 18–23. https://doi.org/10.1145/2676723.2677297

Digital Library

[34]

Jaime Spacco and William Pugh. 2006. Helping Students Appreciate Test-Driven Development (TDD). In Companion to the 21st ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications(OOPSLA ’06). Association for Computing Machinery, New York, NY, USA, 907–913. https://doi.org/10.1145/1176617.1176743

Digital Library

[35]

Claudia Szabo, Nickolas Falkner, Antti Knutas, and Mohsen Dorodchi. 2018. Understanding the Effects of Lecturer Intervention on Computer Science Student Behaviour. In Proceedings of the 2017 ITiCSE Conference on Working Group Reports(ITiCSE-WGR ’17). Association for Computing Machinery, New York, NY, USA, 105–124. https://doi.org/10.1145/3174781.3174787

Digital Library

[36]

Rebecca Vivian, Katrina Falkner, and Nickolas Falkner. 2013. Computer Science Students’ Causal Attributions for Successful and Unsuccessful Outcomes in Programming Assignments. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research(Koli Calling ’13). Association for Computing Machinery, New York, NY, USA, 125–134. https://doi.org/10.1145/2526968.2526982

Digital Library

[37]

Jacqueline L. Whalley and Anne Philpott. 2011. A Unit Testing Approach to Building Novice Programmers’ Skills and Confidence. In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114(ACE ’11). Australian Computer Society, Inc., AUS, 113–118. https://dl.acm.org/doi/10.5555/2459936.2459950

[38]

Wikipedia. 2020. Sokoban – Wikipedia. https://en.wikipedia.org/wiki/Sokoban

[39]

Salla Willman, Rolf Lindén, Erkki Kaila, Teemu Rajala, Mikko-Jussi Laakso, and Tapio Salakoski. 2015. On study habits on an introductory course on programming. Computer Science Education 25, 3 (2015), 276–291. https://doi.org/10.1080/08993408.2015.1073829

Cited By

Domino MShaffer C(2025)Using a wider digital ecosystem to improve self-regulated learningFrontiers in Education10.3389/feduc.2025.148734410Online publication date: 27-Feb-2025
https://doi.org/10.3389/feduc.2025.1487344
Niu JWong JLake CRahardjo JZarkoob HOla OBelleville PMochetti KAllen MMoosvi FWolfman SStone JYuen TShoop LRebelsky SPrather J(2025)Expanding the Horizons of Autograding: Innovative Questions at UBCProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701892(868-874)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701892
Acuña RBansal AMonga MLonati VBarendsen ESheard JPaterson J(2024)Improving Student Learning with Automated AssessmentProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653603(464-470)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653603
Show More Cited By

Index terms have been assigned to the content through auto-classification.

Recommendations

A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

How students are assessed has a powerful effect on their strategies for studying and their learning. When designing assessments, instructors should consider how different approaches for providing feedback to students could encourage positive learning ...
A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to ...
Automated Assessment in Computer Science Education: A State-of-the-Art Review

Practical programming competencies are critical to the success in computer science (CS) education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACE '21: Proceedings of the 23rd Australasian Computing Education Conference

February 2021

195 pages

ISBN:9781450389761

DOI:10.1145/3441636

Editors:
Claudia Szabo
The University of Adelaide, Australia
,
Judy Sheard
Monash University, Australia

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ACE '21

ACE '21: Australasian Computing Education Conference

February 2 - 4, 2021

SA, Virtual, Australia

Acceptance Rates

Overall Acceptance Rate 161 of 359 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Domino MShaffer C(2025)Using a wider digital ecosystem to improve self-regulated learningFrontiers in Education10.3389/feduc.2025.148734410Online publication date: 27-Feb-2025
https://doi.org/10.3389/feduc.2025.1487344
Niu JWong JLake CRahardjo JZarkoob HOla OBelleville PMochetti KAllen MMoosvi FWolfman SStone JYuen TShoop LRebelsky SPrather J(2025)Expanding the Horizons of Autograding: Innovative Questions at UBCProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701892(868-874)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701892
Acuña RBansal AMonga MLonati VBarendsen ESheard JPaterson J(2024)Improving Student Learning with Automated AssessmentProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653603(464-470)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653603
Leinonen JDenny PDorodchi MZhange MCooper S(2024)"Sometimes You Just Gotta Risk It for the Biscuit": A Portrait of Student Risk-TakingProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 110.1145/3649165.3690097(102-108)Online publication date: 5-Dec-2024
https://dl.acm.org/doi/10.1145/3649165.3690097
Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Chiu MMoss ERichards TStephenson BStone JBattestilli LRebelsky SShoop L(2024)Effect of Deadlines on Student Submission Timelines and Success in a Fully-Online Self-Paced CourseProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630837(207-213)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630837
Domino MEdmison BEdwards SMansur RThompson AShaffer C(2024)Identifying critical self-regulated learning skills: a Delphi process studyComputer Science Education10.1080/08993408.2024.2382631(1-28)Online publication date: 27-Oct-2024
https://doi.org/10.1080/08993408.2024.2382631
He DSteffes HYasar SDenny PLuxton-Reilly ALeinonen J(2023)The Forum Factor: Exploring the Link between Online Discourse and Student Achievement in Higher EducationProceedings of the 16th Annual ACM India Compute Conference10.1145/3627217.3627222(80-83)Online publication date: 9-Dec-2023
https://dl.acm.org/doi/10.1145/3627217.3627222
Prather JReeves BDenny PBecker BLeinonen JLuxton-Reilly APowell GFinnie-Ansley JSantos E(2023)“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice ProgrammersACM Transactions on Computer-Human Interaction10.1145/361736731:1(1-31)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1145/3617367
Malone MMonrose FLaakso MMonga MSimon Sheard J(2023)More Carrot or Less Stick: Organically Improving Student Time Management With Practice Tasks and Gamified AssignmentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588825(278-284)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588825
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten