skip to main content
10.1145/3528231.3528357acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

An automatic grading system for a high school-level computational thinking course

Published: 07 February 2023 Publication History

Abstract

Automatic grading systems help lessen the load of manual grading. Most existent autograders are based on unit testing, which focuses on the correctness of the code, but has limited scope for judging code quality. Moreover, it is cumbersome to implement unit testing for evaluating graphical output code. We propose an autograder that can effectively judge the code quality of the visual output codes created by students enrolled in a high school-level computational thinking course. We aim to provide suggestions to teachers on an essential aspect of their grading, namely the level of student competency in using abstraction within their codes. A dataset from five different assignments, including open-ended problems, is used to evaluate the effectiveness of our autograder. Our initial experiments show that our method can classify the students' submissions even for open-ended problems, where existing autograders fail to do so. Additionally, survey responses from course teachers support the importance of our work.

References

[1]
Fernando Alegre, John Underwoood, Juana Moreno, and Mario Alegre. 2020. Introduction to Computational Thinking: A New High School Curriculum Using CodeWorld. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '20). Association for Computing Machinery, New York, NY, USA, 992--998.
[2]
Sara Mernissi Arifi, Ismail Nait Abdellah, Azeddine Zahi, and Rachid Benabbou. 2015. Automatic program assessment using static and dynamic analysis. In 2015 Third World Conference on Complex Systems (WCCS). 1--6.
[3]
David Arthur and Sergei Vassilvitskii. 2006. k-means++: The advantages of careful seeding. Technical Report. Stanford.
[4]
Vera Barstad, Morten Goodwin, and Terje Gjøsæter. 2014. Predicting source code quality with static analysis and machine learning. In Norsk IKT-konferanse for forskning og utdanning.
[5]
Clara Benac Earle, Lars-Åke Fredlund, and John Hughes. 2016. Automatic Grading of Programming Exercises Using Property-Based Testing. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE '16). Association for Computing Machinery, New York, NY, USA, 47--52.
[6]
Guillaume Bouchard and Bill Triggs. 2004. The tradeoff between generative and discriminative classifiers. In 16th IASC International Symposium on Computational Statistics (COMPSTAT'04). 721--728.
[7]
Georg Brandl, Matthäus Chajdas, Armin Ronacher, Pocoo, and Tim Hatch. 2006. Python syntax highlighter. https://pygments.org
[8]
Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). Association for Computing Machinery, New York, NY, USA, 785--794.
[9]
CodeGrade. 2020. Deliver engaging feedback on code. https://www.codegrade.com
[10]
CodeWorld. 2020. Educational computer programming environment using Haskell. https://github.com/google/codeworld
[11]
CSforALL. 2021. CSforAll. https://www.csforall.org/.
[12]
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. ArXiv abs/1810.11363 (2018).
[13]
Stephen H Edwards and Manuel A Perez-Quinones. 2008. Web-CAT: automatically grading programming assignments. In Proceedings of the 13th annual conference on Innovation and technology in computer science education. 328--328.
[14]
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL]
[15]
Chase Geigle, ChengXiang Zhai, and Duncan C Ferguson. 2016. An exploration of automated grading of complex assignments. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 351--360.
[16]
Gradescope. 2020. Online grading platform. https://www.gradescope.com
[17]
Anshul Gupta and Neel Sundaresan. 2018. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'18) Deep Learning Day.
[18]
MH Halstead and T McCabe. 1976. A software complexity measure. IEEE Trans. Software Engineering 2, 12 (1976), 308--320.
[19]
HLint. 2020. Linter for Haskell. https://github.com/ndmitchell/hlint
[20]
Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli calling international conference on computing education research. 86--93.
[21]
Florian Klapproth, Lisa Federkeil, Franziska Heinschke, and Tanja Jungmann. 2020. Teachers' Experiences of Stress and Their Coping Strategies during COVID-19 Induced Distance Teaching. Journal of Pedagogical Research 4, 4 (2020), 444--452.
[22]
LAB.COMPUTER. 2020. Put your computer laboratory on the browser. https://lab.computer
[23]
Xiao Liu, Shuai Wang, Pei Wang, and Dinghao Wu. 2019. Automatic Grading of Programming Assignments: A Formal Semantics Based Approach. In Proceedings 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training ICSE-SEET 2019, 25--31 May 2019, Montreal, Canada.
[24]
Simon Marlow et al. 2010. Haskell 2010 language report. Available online http://www.haskell.org/(May 2011) (2010).
[25]
Dhendra Marutho, Sunarna Hendra Handaka, Ekaprana Wijaya, et al. 2018. The determination of cluster number at k-mean using elbow method and purity evaluation on headline news. In 2018 International Seminar on Application for Technology of Information and Communication. IEEE, 533--538.
[26]
Carnegie Mellon University's School of Computer Science (SCS). 2020. Carnegie Mellon University Computer Science Academy. https://academy.cs.cmu.edu/
[27]
Gabriella Oliveira, Jorge Grenha Teixeira, Ana Torres, and Carla Morais. 2021. An exploratory study on the emergency remote education experience of higher education students and teachers during the COVID-19 pandemic. British Journal of Educational Technology (2021).
[28]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.
[29]
Christopher Piech, Ali Malik, Kylie Jue, and Mehran Sahami. 2021. Code in Place: Online Section Leading for Scalable Human-Centered Learning. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 973--979.
[30]
Joachim Schwieren, Gottfried Vossen, and Peter Westerkamp. 2006. Using Software Testing Techniques for Efficient Handling of Programming Exercises in an e-Learning Platform. Electronic Journal of e-Learning 4, 1 (2006), 87--94.
[31]
Gursimran Singh, Shashank Srikant, and Varun Aggarwal. 2016. Question independent grading using machine learning: The case of computer program grading. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 263--272.
[32]
Shashank Srikant and Varun Aggarwal. 2013. Automatic grading of computer programs: A machine learning approach. In 2013 12th International Conference on Machine Learning and Applications, Vol. 1. IEEE, 85--92.
[33]
Shashank Srikant and Varun Aggarwal. 2014. A System to Grade Computer Programming Skills Using Machine Learning. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, New York, USA) (KDD '14). Association for Computing Machinery, New York, NY, USA, 1887--1896.
[34]
Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. 2021. ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback. arXiv preprint arXiv:2107.14035 (2021).

Cited By

View all
  • (2025)Introducing Code Quality at CS1 Level: Examples and Activities2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709615(339-377)Online publication date: 22-Jan-2025
  • (2024)Evaluating the Potential of Artificial Intelligence in Islamic Religious EducationAI-Enhanced Teaching Methods10.4018/979-8-3693-2728-9.ch010(216-239)Online publication date: 14-Jun-2024
  • (2023)Software Engineering Education for the Next Generation SEENG 2022 Workshop ReportACM SIGSOFT Software Engineering Notes10.1145/3573074.357310448:1(112-116)Online publication date: 17-Jan-2023

Index Terms

  1. An automatic grading system for a high school-level computational thinking course
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          SEENG '22: Proceedings of the 4th International Workshop on Software Engineering Education for the Next Generation
          May 2022
          70 pages
          ISBN:9781450393362
          DOI:10.1145/3528231
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Sponsors

          In-Cooperation

          • IEEE CS

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 07 February 2023

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. code quality
          2. lexical analysis
          3. machine learning
          4. open-ended problems

          Qualifiers

          • Research-article

          Conference

          ICSE '22
          Sponsor:

          Upcoming Conference

          ICSE 2025

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)49
          • Downloads (Last 6 weeks)1
          Reflects downloads up to 10 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2025)Introducing Code Quality at CS1 Level: Examples and Activities2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709615(339-377)Online publication date: 22-Jan-2025
          • (2024)Evaluating the Potential of Artificial Intelligence in Islamic Religious EducationAI-Enhanced Teaching Methods10.4018/979-8-3693-2728-9.ch010(216-239)Online publication date: 14-Jun-2024
          • (2023)Software Engineering Education for the Next Generation SEENG 2022 Workshop ReportACM SIGSOFT Software Engineering Notes10.1145/3573074.357310448:1(112-116)Online publication date: 17-Jan-2023

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media