skip to main content
research-article

Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment Systems

Published: 20 June 2023 Publication History

Abstract

Objectives. Automatic assessment systems are widely used to provide rapid feedback for students and reduce grading time. Despite the benefits of increased efficiency and improved pedagogical outcomes, an ongoing challenge is mitigating poor student behaviors when interacting with automatic assessment systems including numerous submissions, trial-and-error, and relying on marking feedback for problem solving. These behaviors negatively affect student learning as well as have significant impact on system resources. This research quantitatively examines how utilizing submission policies such as limiting the number of submissions and applying regression penalties can reduce negative student behaviors. The hypothesis is that both submission policies will have a significant impact on student behavior and reduce both the number of submissions and regressions in student performance. The research questions evaluate the impact on student behavior, determine which submission policy is the most effective, and what submission policy is preferred by students.
Participants. The study involved two course sections in two different semesters consisting of a total of 224 students at the University of British Columbia, a research-intensive university. The students were evaluated using an automated assessment system in a large third year database course.
Study Methods. The two course sections used an automated assessment system for constructing database design diagrams for assignments and exams. The first section had no limits on the number of submissions for both assignments and exams. The second section had limits for the exams but no limits on assignments. On the midterm, participants were randomly assigned to have either a restriction on the total number of submissions or unlimited submissions but with regression penalties if a graded answer was lower than a previous submission. On the final exam, students were given the option of selecting their submission policy. Student academic performance and submission profiles were compared between the course sections and the different submission policies.
Findings. Unrestricted use of automatic grading systems results in high occurrence of undesirable student behavior including trial-and-error guessing and reduced time between submissions without sufficient independent thought. Both submission policies of limiting maximum submissions and utilizing regression penalties significantly reduce these behaviors by up to 85%. Overall, students prefer maximum submission limits, and demonstrate improved behavior and educational outcomes.
Conclusions. Automated assessment systems when used for larger problems related to design and programming have benefits when deployed with submission restrictions (maximum attempts or regression penalty) for both improved student learning behaviors and to reduce the computational costs for the system. This is especially important for summative assessment but reasonable limits for formative assessments are also valuable.

References

[1]
Kirsti M. Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer Science Education 15, 2 (2005), 83–102.
[2]
Tapio Auvinen. 2015. Harmful study habits in online learning environments with automatic assessment. In 2015 International Conference on Learning and Teaching in Computing and Engineering. 50–57.
[3]
Elisa L. A. Baniassad, Lucas Zamprogno, Braxton Hall, and Reid Holmes. 2021. STOP THE (AUTOGRADER) INSANITY: Regression penalties to deter autograder overreliance. In SIGCSE’21: The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA, March 13-20, 2021, Mark Sherriff, Laurence D. Merkle, Pamela A. Cutter, Alvaro E. Monge, and Judithe Sheard (Eds.). ACM, 1062–1068.
[4]
Weiyi Bian, Omar Alam, and Jörg Kienzle. 2019. Automated grading of class diagrams. In 22nd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion, MODELS Companion 2019, Munich, Germany, September 15–20, 2019. IEEE, 700–709.
[5]
Weiyi Bian, Omar Alam, and Jörg Kienzle. 2020. Is automated grading of models effective? Assessing automated grading of class diagrams. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS’20). ACM, New York, NY, USA, 365–376.
[6]
Paul Black and Dylan Wiliam. 2009. Developing the theory of formative assessment. Educational Assessment Evaluation and Accountability 21 (022009), 5–31.
[7]
Benjamin S. Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher 13, 6 (1984), 4–16.
[8]
Younes Boubekeur, Gunter Mussbacher, and Shane McIntosh. 2020. Automatic assessment of students’ software models using a simple heuristic and machine learning. In MODELS’20: ACM/IEEE 23rd International Conference on Model Driven Engineering Languages and Systems, Canada, 18-23 October, 2020, Companion Proceedings. ACM, 20:1–20:10.
[9]
Kevin Buffardi and Stephen H. Edwards. 2015. Reconsidering automated feedback: A test-driven approach. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE 2015, Kansas City, MO, USA, March 4-7, 2015. ACM, 416–420.
[10]
Sébastien Combéfis. 2022. Automated code assessment for education: Review, classification and perspectives on techniques and tools. Software 1, 1 (Feb.2022), 3–30.
[11]
Stephen H. Edwards. 2004. Using software testing to move students from trial-and-error to reflection-in-action. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’04). Association for Computing Machinery, New York, NY, USA, 26–30.
[12]
Stephen H. Edwards and Manuel A. Pérez-Quiñones. 2008. Web-CAT: Automatically grading programming assignments. In Proceedings of the 13th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE 2008. ACM, 328.
[13]
Emily Faulconer, J. C. Griffith, and H. Frank. 2021. If at first you do not succeed: Student behavior when provided feedforward with multiple trials for online summative assessments. Teaching in Higher Education 26, 4 (2021), 586–601.
[14]
Sarah Foss, Tatiana Urazova, and Ramon Lawrence. 2022. Automatic generation and marking of UML database design diagrams. In SIGCSE 2022: The 53rd ACM Technical Symposium on Computer Science Education, Providence, RI, USA, March 3-5, 2022, Volume 1, Larry Merkle, Maureen Doyle, Judithe Sheard, Leen-Kiat Soh, and Brian Dorn (Eds.). ACM, 626–632.
[15]
Sarah Foss, Tatiana Urazova, and Ramon Lawrence. 2022. Learning UML database design and modeling with AutoER. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, MODELS 2022, Montreal, Quebec, Canada, October 23-28, 2022, Thomas Kühn and Vasco Sousa (Eds.). ACM, 42–45.
[16]
Maxwell Fowler, Binglin Chen, Sushmita Azad, Matthew West, and Craig B. Zilles. 2021. Autograding explain in plain English questions using NLP. In SIGCSE’21: The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA, March 13-20, 2021. ACM, 1163–1169.
[17]
Object Management Group. 2017. OMG® Unified Modeling Language® (OMG UML®). (2017). Retrieved: 2023-02-01. https://www.omg.org/spec/UML/2.5.1/
[18]
Marcelo Guerra Hahn, Silvia Margarita Baldiris Navarro, Luis De La Fuente Valentín, and Daniel Burgos. 2021. A systematic review of the effects of automatic scoring and automatic feedback in educational settings. IEEE Access (2021).
[19]
Georgiana Haldeman, Monica Babes-Vroman, Andrew Tjang, and Thu D. Nguyen. 2021. CSF: Formative feedback in autograding. ACM Trans. Comput. Educ. 21, 3 (2021), 21:1–21:30.
[20]
Robert W. Hasker. 2011. UMLGrader: An automated class diagram grader. Journal of Computing Sciences in Colleges 27, 1 (2011), 47–54.
[21]
Robert W. Hasker and Mike Rowe. 2011. UMLint: Identifying defects in UML diagrams. In 2011 ASEE Annual Conference & Exposition. 22.1558.1–22.1558.14.
[22]
Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling’10). Association for Computing Machinery, New York, NY, USA, 86–93.
[23]
Michael S. Irwin and Stephen H. Edwards. 2019. Can mobile gaming psychology be used to improve time management on programming assignments?. In Proceedings of the ACM Conference on Global Computing Education (CompEd’19). Association for Computing Machinery, 208–214.
[24]
Ville Karavirta, Ari Korhonen, and Lauri Malmi. 2006. On the use of resubmissions in automatic assessment systems. Computer Science Education 16, 3 (2006), 229–240.
[25]
Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Trans. Comput. Educ. 19, 1, Article 3 (Sep.2018), 43 pages.
[26]
Haden Hooyeon Lee. 2021. Effectiveness of real-time feedback and instructive hints in graduate CS courses via automated grading system. In SIGCSE’21: The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA, March 13-20, 2021. ACM, 101–107.
[27]
Xiao Liu, Shuai Wang, Pei Wang, and Dinghao Wu. 2019. Automatic grading of programming assignments: An approach based on formal semantics. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 126–137.
[28]
Madeleine Lorås, Guttorm Sindre, Hallvard Trætteberg, and Trond Aalberg. 2021. Study behavior in computing education—a systematic literature review. ACM Trans. Comput. Educ. 22, 1, Article 9 (Oct.2021), 40 pages.
[29]
Lauri Malmi, Ville Karavirta, Ari Korhonen, and Jussi Nikander. 2005. Experiences on automatically assessed algorithm simulation exercises with different resubmission policies. J. Educ. Resour. Comput. 5, 3 (Sep.2005), 7–es.
[30]
Marina Marchisio, Alice Barana, Michele Fioravera, Sergio Rabellino, and Alberto Conte. 2018. A model of formative automatic assessment and interactive feedback for STEM. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 01. 1016–1025.
[31]
Samiha Marwan, Anay Dombe, and Thomas W. Price. 2020. Unproductive help-seeking in programming: What it is and how to address it. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE’20). Association for Computing Machinery, New York, NY, USA, 54–60.
[32]
Samiha Marwan, Ge Gao, Susan Fisk, Thomas W. Price, and Tiffany Barnes. 2020. Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In Proceedings of the 2020 ACM Conference on International Computing Education Research (ICER’20). Association for Computing Machinery, New York, NY, USA, 194–203.
[33]
Jessica McBroom, Irena Koprinska, and Kalina Yacef. 2021. A survey of automated programming hint generation: The HINTS framework. ACM Comput. Surv. 54, 8, Article 172 (Oct.2021), 27 pages.
[34]
Igor Mekterović, Ljiljana Brkić, Boris Milašinović, and Mirta Baranović. 2020. Building a comprehensive automated programming assessment system. IEEE Access 8 (2020), 81154–81172.
[35]
Raymond Pettit, John Homer, Roger Gee, Susan Mengel, and Adam Starbuck. 2015. An empirical study of iterative improvement in programming assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE’15). Association for Computing Machinery, New York, NY, USA, 410–415.
[36]
Vreda Pieterse. 2013. Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research (CSERC’13). Open Universiteit, Heerlen, Heerlen, NLD, 45–56.
[37]
Josep Soler, Imma Boada, Ferran Prados, Jordi Poch, and Ramón Fabregat. 2010. A formative assessment tool for conceptual database design using UML class diagram. iJET 5, 3 (2010), 27–33. https://www.online-journals.org/index.php/i-jet/article/view/1402.
[38]
Luciane Telinski, Wiedermann Agner, and Timothy C. Lethbridge. 2017. A survey of tool use in modeling education. In 20th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, MODELS 2017, Austin, TX, USA, September 17-22, 2017. IEEE Computer Society, 303–311.
[39]
Luciane Telinski, Wiedermann Agner, Timothy C. Lethbridge, and Inali Wisniewski Soares. 2019. Student experience with software modeling tools. Softw. Syst. Model. 18, 5 (2019), 3025–3047.
[40]
Pete G. Thomas, Kevin G. Waugh, and Neil Smith. 2006. Using patterns in the automatic marking of ER-diagrams. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE 2006, Bologna, Italy, June 26-28, 2006. ACM, 83–87.
[41]
Matthew West, Geoffrey L. Herman, and C. Zilles. 2015. PrairieLearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 122nd ASEE Annual Conference and Exposition. 26.1238.1–26.1238.14.
[42]
Bojin Yao, Qitian Liao, Connor McMahon, and Daniel D. Garcia. 2021. Formal categorization of variants for question generators in computer-based assessments. In SIGCSE’21: The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA, March 13-20, 2021. ACM, 1244.

Index Terms

  1. Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment Systems

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Computing Education
        ACM Transactions on Computing Education  Volume 23, Issue 3
        September 2023
        233 pages
        EISSN:1946-6226
        DOI:10.1145/3605196
        • Editor:
        • Amy J. Ko
        Issue’s Table of Contents

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 20 June 2023
        Online AM: 20 April 2023
        Accepted: 31 March 2023
        Revised: 27 January 2023
        Received: 23 June 2022
        Published in TOCE Volume 23, Issue 3

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Automatic marking
        2. auto-grading
        3. assessment
        4. submission limits
        5. regression penalties
        6. programming and design questions
        7. UML
        8. database

        Qualifiers

        • Research-article

        Funding Sources

        • UBC Okanagan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 589
          Total Downloads
        • Downloads (Last 12 months)202
        • Downloads (Last 6 weeks)21
        Reflects downloads up to 18 Jan 2025

        Other Metrics

        Citations

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        Full Text

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media