short-paper

PST: Measuring Skill Proficiency in Programming Exercise Process via Programming Skill Tracing

Authors:

Shuanghong Shen,

Enhong ChenAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2601 - 2606

https://doi.org/10.1145/3477495.3531903

Published: 07 July 2022 Publication History

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

PST: Measuring Skill Proficiency in Programming Exercise Process via Programming Skill Tracing

Pages 2601 - 2606

Abstract
Supplementary Material
References

Abstract

Programming has become an important skill for individuals nowadays. For the demand to improve personal programming skill, tracking programming skill proficiency is getting more and more important. However, few researchers pay attention to measuring the programming skill of learners. Most of existing studies on learner capability portrait only made use of the exercise results, while the rich behavioral information contained in programming exercise process remains unused. Therefore, we propose a model that measures skill proficiency in programming exercise process named Programming Skill Tracing (PST). We designed Code Information Graph (CIG) to represent the feature of learners' solution code, and Code Tracing Graph (CTG) to measure the changes between the adjacent submissions. Furthermore, we divided programming skill into programming knowledge and coding ability to get more fine-grained assessment. Finally, we conducted various experiments to verify the effectiveness and interpretability of our PST model.

Supplementary Material

MP4 File (SIGIR22-sp2088.mp4)

Presentation video for paper PST: Measuring Skill Proficiency in Programming Exercise Process via Programming Skill Tracing

Download
8.65 MB

References

[1]

Ghodai Abdelrahman and Qing Wang. 2019. Knowledge tracing with sequential key-value memory networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 175--184.

Digital Library

[2]

Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: A learnable representation of code semantics. arXiv preprint arXiv:1806.07336 (2018).

[3]

Marc Berges, Andreas Mühling, and Peter Hubwieser. 2012. The gap between knowledge and ability. In Proceedings of the 12th Koli Calling international conference on computing education research. 126--134.

Digital Library

[4]

Karo Castro-Wunsch, Alireza Ahadi, and Andrew Petersen. 2017. Evaluating neural networks as a method for identifying students in need of assistance. In Proceedings of the 2017 ACM SIGCSE technical symposium on computer science education. 111--116.

Digital Library

[5]

Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253--278.

[6]

Susan E Embretson and Steven P Reise. 2013. Item response theory. Psychology Press.

[7]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).

[8]

Ge Gao, Samiha Marwan, and Thomas W Price. 2021. Early Performance Prediction using Interpretable Patterns in Programming Process Data. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 342--348.

Digital Library

[9]

Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. 2021. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 501--510.

Digital Library

[10]

Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020).

[11]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.

Digital Library

[12]

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2073--2083.

[13]

Tanja Käser, Severin Klingler, Alexander G Schwing, and Markus Gross. 2017. Dynamic Bayesian networks for student modeling. IEEE Transactions on Learning Technologies 10, 4 (2017), 450--462.

Digital Library

[14]

Jussi Kasurinen and Uolevi Nikula. 2009. Estimating programming knowledge with Bayesian knowledge tracing. ACM SIGCSE Bulletin 41, 3 (2009), 313--317.

Digital Library

[15]

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015).

[16]

Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1 (2019), 100--115.

Digital Library

[17]

M Lu, Y Wang, D Tan, and L Zhao. 2021. Student Program Classification Using Gated Graph Attention Neural Network. IEEE Access (2021).

[18]

Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. arXiv preprint arXiv:1506.05908 (2015).

[19]

Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. arXiv preprint arXiv:2105.12655 (2021).

[20]

Mark D Reckase. 2009. Multidimensional item response theory models. In Multidimensional item response theory. Springer, 79--112.

[21]

Cristóbal Romero, Sebastián Ventura, Pedro G Espejo, and César Hervás. 2008. Data mining algorithms to classify students. In Educational data mining 2008.

[22]

Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, and Shijin Wang. 2021. Learning Process-consistent Knowledge Tracing. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1452--1460.

Digital Library

[23]

Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, and Shijin Wang. 2020. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1857--1860.

Digital Library

[24]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929--1958.

Digital Library

[25]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).

[26]

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. 2020. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6153--6161.

[27]

Lisa Wang, Angela Sy, Larry Liu, and Chris Piech. 2017. Deep knowledge tracing on programming exercises. In Proceedings of the fourth (2017) ACM conference on learning@ scale. 201--204.

Digital Library

[28]

Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 261--271.

[29]

Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 783--794.

Digital Library

[30]

Renyu Zhu, Dongxiang Zhang, Chengcheng Han, Ming Gao, Xuesong Lu, Weining Qian, and Aoying Zhou. 2021. Programming Knowledge Tracing: A Comprehensive Dataset and A New Model. arXiv preprint arXiv:2112.08273 (2021).

Cited By

Gao ZCui CYan HLiu JSun XFeng JStone JYuen TShoop LRebelsky SPrather J(2025)Towards a Quantitative Competency Model for CS1 via Five-Channel Learning SequencesProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701837(367-373)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701837
Sun XLiu QZhang KShen SZhuang YGuo Y(2025)LGS-KT: Integrating logical and grammatical skills for effective programming knowledge tracingNeural Networks10.1016/j.neunet.2025.107164185(107164)Online publication date: May-2025
https://doi.org/10.1016/j.neunet.2025.107164
Lin XHuang ZZhao HChen ELiu QLian DLi XWang H(2024)Learning Relation-Enhanced Hierarchical Solver for Math Word ProblemsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.327211435:10(13830-13844)Online publication date: Oct-2024
https://doi.org/10.1109/TNNLS.2023.3272114
Show More Cited By

Index Terms

PST: Measuring Skill Proficiency in Programming Exercise Process via Programming Skill Tracing
1. Computing methodologies
  1. Artificial intelligence
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Enhancing Programming Knowledge Tracing by Interacting Programming Skills and Student Code
LAK22: LAK22: 12th International Learning Analytics and Knowledge Conference

Programming education has received extensive attention in recent years due to the increasing demand for programming ability in almost all industries. Educational institutions have widely employed online judges for programming training, which can help ...
A Metrics-Based Model for Programming Skill
ICER '20: Proceedings of the 2020 ACM Conference on International Computing Education Research

Programming is a core skill in computing. Much of computing education and work in industry and research has programming skill is an important factor. Human subjects have varying levels of programming experience which affects their output. The inability ...
Survey for Smart and Adaptative Education
ICETM '22: Proceedings of the 2022 5th International Conference on Education Technology Management

Technological media has become a vital component for educators, enabling greater efficiency in lesson planning and preparation, providing active repositories of student work, as well as more personalised learning for students. The construction of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

National Key Research and Development Program of China
National Natural Science Foundation of China

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
317
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)12

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gao ZCui CYan HLiu JSun XFeng JStone JYuen TShoop LRebelsky SPrather J(2025)Towards a Quantitative Competency Model for CS1 via Five-Channel Learning SequencesProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701837(367-373)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701837
Sun XLiu QZhang KShen SZhuang YGuo Y(2025)LGS-KT: Integrating logical and grammatical skills for effective programming knowledge tracingNeural Networks10.1016/j.neunet.2025.107164185(107164)Online publication date: May-2025
https://doi.org/10.1016/j.neunet.2025.107164
Lin XHuang ZZhao HChen ELiu QLian DLi XWang H(2024)Learning Relation-Enhanced Hierarchical Solver for Math Word ProblemsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.327211435:10(13830-13844)Online publication date: Oct-2024
https://doi.org/10.1109/TNNLS.2023.3272114
Liu YZhu RGao M(2024)Personalized Programming Guidance Based on Deep Programming Learning Style CapturingComputer Science and Education. Computer Science and Technology10.1007/978-981-97-0730-0_20(214-231)Online publication date: 26-Feb-2024
https://doi.org/10.1007/978-981-97-0730-0_20
Li GShuai JHu YZhang YWang YYang TXiong N(2022)DKT-LCIRT: A Deep Knowledge Tracking Model Integrating Learning Capability and Item Response TheoryElectronics10.3390/electronics1120336411:20(3364)Online publication date: 18-Oct-2022
https://doi.org/10.3390/electronics11203364

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten