Nynne Grauslund Kristiansen
ABSTRACT
Existing research does not quantify and compare the differences between automated and manual assessment in the context of feedback on programming assignments. This makes it hard to reason about the effects of adopting automated assessment at the expense of manual assessment. Based on a controlled experiment involving N=117 undergraduate first-semester CS1 students, we compare the effects of having access to feedback from: i) only automated assessment, ii) only manual assessment (in the form of teaching assistants), and iii) both automated as well as manual assessment. The three conditions are compared in terms of (objective) task effectiveness and from a (subjective) student perspective.
The experiment demonstrates that having access to both forms of assessment (automated and manual) is superior both from a task effectiveness as well as a student perspective. We also find that the two forms of assessment are complementary: automated assessment appears to be better in terms of task effectiveness; whereas manual assessment appears to be better from a student perspective. Further, we found that automated assessment appears to be working better for men than women, who are significantly more inclined towards manual assessment. We then perform a cost/benefit analysis which leads to the identification of four equilibria that appropriately balance costs and benefits. Finally, this gives rise to four recommendations of when to use which kind or combination of feedback (manual and/or automated), depending on the number of students and the amount of per-student resources available. These observations provide educators with evidence-based justification for budget requests and considerations on when to (not) use automated assessment.
- [n. d.]. University Teaching Assistant Salaries by Country. https://www.salaryexpert.com/salary/browse/countries/university-teaching-assistant. Accessed: 2023-03-17.Google Scholar
- Aditi Agrawal and Benjamin Reed. 2022. A survey on grading format of automated grading tools for programming assignments. arXiv preprint arXiv:2212.01714 (2022).Google Scholar
- Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education 15, 2 (2005), 83–102.Google Scholar
- José Luis Fernández Alemán. 2010. Automated assessment in a programming tools course. IEEE Transactions on Education 54, 4 (2010), 576–581.Google ScholarDigital Library
- Joe Michael Allen, Frank Vahid, Kelly Downey, and Alex Daniel Edgcomb. 2018. Weekly programs in a CS1 class: Experiences with auto-graded many-small programs (MSP). In 2018 ASEE Annual Conference & Exposition.Google ScholarCross Ref
- Enrique Barra, Sonsoles López-Pernas, Álvaro Alonso, Juan Fernando Sánchez-Rada, Aldo Gordillo, and Juan Quemada. 2020. Automated assessment in programming courses: A case study during the COVID-19 era. Sustainability 12, 18 (2020), 7451.Google ScholarCross Ref
- Julio C Caiza and José María del Álamo Ramiro. 2013. Programming assignments automatic grading: review of tools and implementations. (2013).Google Scholar
- Ingrid Maria Christensen, Melissa Høegh Marcher, Paweł Grabarczyk, Therese Graversen, and Claus Brabrand. 2021. Computing Educational Activities Involving People Rather Than Things Appeal More to Women (Recruitment Perspective). In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 127–144. https://doi.org/10.1145/3446871.3469758Google ScholarDigital Library
- Paul E. Dickson, Toby Dragon, and Adam Lee. 2017. Using Undergraduate Teaching Assistants in Small Classes. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (Seattle, Washington, USA) (SIGCSE ’17). Association for Computing Machinery, New York, NY, USA, 165–170.Google ScholarDigital Library
- Dante D Dixson and Frank C Worrell. 2016. Formative and summative assessment in the classroom. Theory into practice 55, 2 (2016), 153–159.Google Scholar
- Stephen H Edwards. 2003. Using test-driven development in the classroom: Providing students with automatic, concrete feedback on performance. In Proceedings of the international conference on education and information systems: technologies and applications EISTA, Vol. 3. Citeseer.Google Scholar
- Emma Enström, Gunnar Kreitz, Fredrik Niemelä, Pehr Söderman, and Viggo Kann. 2011. Five years with kattis—using an automated assessment system in teaching. In 2011 Frontiers in education conference (FIE). IEEE, T3J–1.Google Scholar
- Peter Farrell, Alison Alborz, Andy Howes, and Diana Pearson. 2010. The impact of teaching assistants on improving pupils’ academic achievement in mainstream schools: A review of the literature. Educational review 62, 4 (2010), 435–448.Google Scholar
- Bent Flyvbjerg. 2006. Five misunderstandings about case-study research. Qualitative inquiry 12, 2 (2006), 219–245.Google Scholar
- Jeffrey Forbes, David J. Malan, Heather Pon-Barry, Stuart Reges, and Mehran Sahami. 2017. Scaling Introductory Courses Using Undergraduate Teaching Assistants. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (Seattle, Washington, USA) (SIGCSE ’17). Association for Computing Machinery, New York, NY, USA, 657–658.Google ScholarDigital Library
- Adam M Gaweda and Collin F Lynch. 2021. Student Practice Sessions Modeled as ICAP Activity Silos.International Educational Data Mining Society (2021).Google Scholar
- Imran Ghory. 2007. Using FizzBuzz to Find Developers who Grok Coding. https://imranontech.com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/. Accessed: 2023-01-13.Google Scholar
- Aldo Gordillo. 2019. Effect of an instructor-centered tool for automatic assessment of programming assignments on students’ perceptions and performance. Sustainability 11, 20 (2019), 5568.Google ScholarCross Ref
- Pawel Grabarczyk, Alma Freiesleben, Amanda Bastrup, and Claus Brabrand. 2022. Computing Educational Programmes with More Women Are More about People & Less about Things. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 (Dublin, Ireland) (ITiCSE ’22). Association for Computing Machinery, New York, NY, USA, 172–178. https://doi.org/10.1145/3502718.3524784Google ScholarDigital Library
- Pawel Grabarczyk, Sebastian Mateos Nicolajsen, and Claus Brabrand. 2022. On the Effect of Onboarding Computing Students without Programming-Confidence or -Experience. In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’22). Association for Computing Machinery, New York, NY, USA, Article 18, 8 pages. https://doi.org/10.1145/3564721.3564724Google ScholarDigital Library
- Qiang Hao, David H Smith IV, Lu Ding, Amy Ko, Camille Ottaway, Jack Wilson, Kai H Arakawa, Alistair Turcan, Timothy Poehlman, and Tyler Greer. 2022. Towards understanding the effective design of automated formative feedback for programming assignments. Computer Science Education 32, 1 (2022), 105–127.Google ScholarCross Ref
- Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli calling international conference on computing education research. 86–93.Google ScholarDigital Library
- David Insa and Josep Silva. 2018. Automatic assessment of Java code. Computer Languages, Systems & Structures 53 (2018), 59–72. https://doi.org/10.1016/j.cl.2018.01.004Google ScholarCross Ref
- Code judge. 2023. Code judge. https://codejudge.io. Accessed: 2023-01-13.Google Scholar
- Melissa Høegh Marcher, Ingrid Maria Christensen, Paweł Grabarczyk, Therese Graversen, and Claus Brabrand. 2021. Computing Educational Activities Involving People Rather Than Things Appeal More to Women (CS1 Appeal Perspective). In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 145–156. https://doi.org/10.1145/3446871.3469761Google ScholarDigital Library
- Dragan Mirković and S Lennart Johnsson. 2003. CODELAB: A Developers’ Tool for Efficient Code Generation and Optimization. In International Conference on Computational Science. Springer, 729–738.Google Scholar
- Diba Mirza, Phillip T Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate teaching assistants in computer science: a systematic literature review. In Proceedings of the 2019 ACM Conference on International Computing Education Research. 31–40.Google ScholarDigital Library
- Stephen Nutbrown and Colin Higgins. 2016. Static analysis of programming exercises: Fairness, usefulness and a method for application. Computer Science Education 26, 2-3 (2016), 104–128.Google ScholarCross Ref
- José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Trans. Comput. Educ. 22, 3, Article 34 (jun 2022), 40 pages. https://doi.org/10.1145/3513140Google ScholarDigital Library
- José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated assessment in computer science education: A state-of-the-art review. ACM Transactions on Computing Education (TOCE) 22, 3 (2022), 1–40.Google ScholarDigital Library
- Laura Pappano. 2012. The Year of the MOOC. The New York Times 2, 12 (2012), 2012.Google Scholar
- Raymond Scott Pettit, John D Homer, Kayla Michelle McMurry, Nevan Simone, and Susan A Mengel. 2015. Are automated assessment tools helpful in programming courses?. In 2015 ASEE Annual Conference & Exposition. 26–230.Google ScholarCross Ref
- Vreda Pieterse and Janet Liebenberg. 2017. Automatic vs Manual Assessment of Programming Tasks. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’17). Association for Computing Machinery, New York, NY, USA, 193–194. https://doi.org/10.1145/3141880.3141912Google ScholarDigital Library
- Emma Riese and Viggo Kann. 2020. Teaching assistants’ experiences of tutoring and assessing in computer science education. In 2020 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9.Google ScholarDigital Library
- Jonathan Sharples, P Blatchford, and R Webster. 2016. Making best use of teaching assistants. (2016).Google Scholar
- Janet Siegmund, Norbert Siegmund, and Sven Apel. 2015. Views on internal and external validity in empirical software engineering. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 9–19.Google ScholarCross Ref
- SonarQube. 2023. SonarQube Documentation. https://docs.sonarqube.org/latest/. Accessed: 2023-01-13.Google Scholar
- Thomas Staubitz, Hauke Klement, Jan Renz, Ralf Teusner, and Christoph Meinel. 2015. Towards practical programming exercises and automated assessment in Massive Open Online Courses. In 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). IEEE, 23–30.Google ScholarCross Ref
- Zahid Ullah, Adidah Lajis, Mona Jamjoom, Abdulrahman Altalhi, Abdullah Al-Ghamdi, and Farrukh Saleem. 2018. The effect of automatic assessment on novice programming: Strengths and limitations of existing systems. Computer Applications in Engineering Education 26, 6 (2018), 2328–2341.Google ScholarCross Ref
- Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist 46, 4 (2011), 197–221.Google Scholar
- Chris Wilcox. 2015. The role of automation in undergraduate computer science education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 90–95.Google ScholarDigital Library
Index Terms
- Feedback on Student Programming Assignments: Teaching Assistants vs Automated Assessment Tool
Recommendations
Competition and Feedback through Automated Assessment in a Data Structures and Algorithms Course
ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science EducationWe have investigated competitive elements and different forms of feedback through automated assessment in a Data Structures and Algorithms course. It is given at the start of the second year and has about 140 students. In 2011 we investigated the ...
Automated Assessment: Experiences From the Trenches
ACE '23: Proceedings of the 25th Australasian Computing Education ConferenceAutomated assessment is commonly used across the spectrum of computing courses offered by Tertiary institutions. Such assessment is frequently intended to address the scalability of feedback that is essential for learning, and assessment for ...
Automated Assessment of Programming Assignments
CSERC '13: Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education ResearchThis is a position paper in which I argue that massive open online programming courses can benefit by the application of automated assessment of programming assignments.
I gathered success factors and identified concerns related to automatic assessment ...
Comments