research-article

What different kinds of stratification can reveal about the generalizability of data-mined skill assessment models

Authors:

Michael A. Sao Pedro,

Ryan S. J. D. Baker,

Janice D. GobertAuthors Info & Claims

LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge

Pages 190 - 194

https://doi.org/10.1145/2460296.2460334

Published: 08 April 2013 Publication History

Get Access

Abstract

When validating assessment models built with data mining, generalization is typically tested at the student-level, where models are tested on new students. This approach, though, may fail to find cases where model performance suffers if other aspects of those cases relevant to prediction are not well represented. We explore this here by testing if scientific inquiry skill models built and validated for one science topic can predict skill demonstration for new students and a new science topic. Test cases were chosen using two methods: student-level stratification, and stratification based on the amount of trials ran during students' experimentation. We found that predictive performance of the models was different on each test set, revealing limitations that would have been missed from student-level validation alone.

References

[1]

Baker, R. S. J. D, Corbett, A. T., Roll, I., and Koedinger, K. R. Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3 (2008), 287--314.

Digital Library

Google Scholar

[2]

Blikstein, P. Using Learning Analytics to Assess Students' Behavior in Open-Ended Programming Tasks. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (Banff, AB, CA 2011), 110--116.

Digital Library

Google Scholar

[3]

Ghazarian, A. and Noorhosseini, S. M. Automatic Detection of Users' Skill Levels Using High-Frequency User Interface Events. User Modeling and User-Adapted Interaction, 20, 2 (2010), 109--146.

Digital Library

Google Scholar

[4]

Sao Pedro, M. A., Baker, R. S. J. D., Gobert, J. D., Montalvo, O., and Nakama, A. Leveraging Machine-Learned Detectors of Systematic Inquiry Behavior to Estimate and Predict Transfer of Inquiry Skill. User Modeling and User-Adapted Interaction (2011), 1--39.

Digital Library

Google Scholar

[5]

Siemens, G. Learning Analytics: Envisioning a Research Discipline and a Domain of Practice. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (Banff, AB, CA 2012), ACM, 4--8.

Digital Library

Google Scholar

[6]

Pardos, Z., Baker, R., Gowda, S., and Heffernan, N. The Sum is Greater than the Parts: Ensembling Models of Student Knowledge in Educational Software. SIGKDD Explorations, 13, 2 (2011), 37--44.

Digital Library

Google Scholar

[7]

Sao Pedro, M., Baker, R., and Gobert, J. Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. In Proceedings of the 20th Conference on User Modeling, Adaptation, and Personalization (Montreal, QC, Canada 2012), 249--260.

Digital Library

Google Scholar

[8]

Gobert, J., Sao Pedro, M., Baker, R., Toto, E., and Montalvo, O. Leveraging educational data mining for real time performance assessment of scientific inquiry skills within microworlds. Journal of Educational Data Mining, 4, 1 (2012), 111--143.

Google Scholar

[9]

Baker, R. S. J. D., Corbett, A. T., and Wagner, A. Z. Human Classification of Low-Fidelity Replays of Student Actions. In Proceedings of the Educational Data Mining Workshop held at the 8th International Conference on Intelligent Tutoring Systems, ITS 2006 (Jhongli, Taiwan 2006), 29--36.

Google Scholar

[10]

Hanley, J. A. and McNeil, B. J. The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology, 143 (1982), 29--36.

Crossref

Google Scholar

Cited By

View all

Ayanwale MAdelana OOdufuwa T(2024)Exploring STEAM teachers’ trust in AI-based educational technologies: a structural equation modelling approachDiscover Education10.1007/s44217-024-00092-z3:1Online publication date: 26-Apr-2024
https://doi.org/10.1007/s44217-024-00092-z
Giannakos MGiannakos M(2022)Issues to Consider as a CCI and Learning Technology ResearcherExperimental Studies in Learning Technology and Child–Computer Interaction10.1007/978-3-031-14350-2_10(97-104)Online publication date: 1-Oct-2022
https://doi.org/10.1007/978-3-031-14350-2_10
Salehian Kia FHatala MBaker RTeasley SScheffel MDowell NJoksimovic SSiemens G(2021)Measuring Students’ Self-Regulatory Phases in LMS with Behavior and Real-Time Self ReportLAK21: 11th International Learning Analytics and Knowledge Conference10.1145/3448139.3448164(259-268)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.1145/3448139.3448164
Show More Cited By

Index Terms

What different kinds of stratification can reveal about the generalizability of data-mined skill assessment models
1. Applied computing
  1. Education
    1. Computer-assisted instruction
    2. Computer-managed instruction
  2. Enterprise computing
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Examining change in students’ self-regulated learning patterns after a formative assessment using process mining techniques
Abstract
The growing popularity of e-learning platforms, such as learning management systems, has foregrounded the role of self-regulated learning (SRL) in student success. In many e-learning environments, students typically complete learning assignments ...
Highlights
- Students engage in a loosely sequenced recursive self-regulated learning cycle.
- Students actively adjust self-regulated learning patterns in response to a formative assessment.
- There are group differences in the adjustment of self-...
On the role of gamification and localization in an open online learning environment: javala experiences
Koli Calling '15: Proceedings of the 15th Koli Calling Conference on Computing Education Research

Massive Open Online Courses (MOOCs) have rapidly become an important tool for educational institutes in teaching programming. Nevertheless, high drop-out rates have always been a problem in online learning. As MOOCs have become an important part of ...
A supervised learning framework for learning management systems
DATA '18: Proceedings of the First International Conference on Data Science, E-learning and Information Systems

Educational Data Mining (EDM) and Learning Analytics (LA) focus on data analysis of learners in the context of educational settings like Moodle, a Learning Management System (LMS). Both EDM and LA aim to understand learners and optimise learning ...

Comments

Information & Contributors

Information

Published In

LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge

April 2013

300 pages

ISBN:9781450317856

DOI:10.1145/2460296

Editors:
Dan Suthers
University of Hawaii, USA
,
Katrien Verbert
Eindhoven University of Technology, The Netherlands
,
Erik Duval
University of Leuven, Belgium
,
Xavier Ochoa
Escuela Superior Politécnica del Litoral, Ecuador

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

LAK '13

Sponsor:

SIGWEB
K.U. Leuven
SIGCHI

LAK '13: Third Conference on Learning Analytics and Knowledge

April 8 - 13, 2013

Leuven, Belgium

Acceptance Rates

LAK '13 Paper Acceptance Rate 16 of 58 submissions, 28%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
320
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ayanwale MAdelana OOdufuwa T(2024)Exploring STEAM teachers’ trust in AI-based educational technologies: a structural equation modelling approachDiscover Education10.1007/s44217-024-00092-z3:1Online publication date: 26-Apr-2024
https://doi.org/10.1007/s44217-024-00092-z
Giannakos MGiannakos M(2022)Issues to Consider as a CCI and Learning Technology ResearcherExperimental Studies in Learning Technology and Child–Computer Interaction10.1007/978-3-031-14350-2_10(97-104)Online publication date: 1-Oct-2022
https://doi.org/10.1007/978-3-031-14350-2_10
Salehian Kia FHatala MBaker RTeasley SScheffel MDowell NJoksimovic SSiemens G(2021)Measuring Students’ Self-Regulatory Phases in LMS with Behavior and Real-Time Self ReportLAK21: 11th International Learning Analytics and Knowledge Conference10.1145/3448139.3448164(259-268)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.1145/3448139.3448164
Luan HGeczy PLai HGobert JYang SOgata HBaltes JGuerra RLi PTsai C(2020)Challenges and Future Directions of Big Data and Artificial Intelligence in EducationFrontiers in Psychology10.3389/fpsyg.2020.58082011Online publication date: 19-Oct-2020
https://doi.org/10.3389/fpsyg.2020.580820
Smith ALeeman-Munk SShelton AMott BWiebe ELester J(2019)A Multimodal Assessment Framework for Integrating Student Writing and Drawing in Elementary Science LearningIEEE Transactions on Learning Technologies10.1109/TLT.2018.279987112:1(3-15)Online publication date: 1-Jan-2019
https://doi.org/10.1109/TLT.2018.2799871
Martínez-Cerdá JTorrent-Sellens J(2018)Undirected Bipartite Networks as an Alternative Methodology to Probabilistic ExplorationGraph Theoretic Approaches for Analyzing Large-Scale Social Networks10.4018/978-1-5225-2814-2.ch005(75-94)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-2814-2.ch005
Gobert JMoussavi RLi HSao Pedro MDickler R(2018)Real-Time Scaffolding of Students’ Online Data Interpretation During Inquiry with Inq-ITS Using Educational Data MiningCyber-Physical Laboratories in Engineering and Science Education10.1007/978-3-319-76935-6_8(191-217)Online publication date: 27-Apr-2018
https://doi.org/10.1007/978-3-319-76935-6_8
Leeman-Munk SWiebe ELester JPistilli MWillis JKoch DArnold KTeasley SPardo A(2014)Assessing elementary students' science competency with text analyticsProceedings of the Fourth International Conference on Learning Analytics And Knowledge10.1145/2567574.2567620(143-147)Online publication date: 24-Mar-2014
https://dl.acm.org/doi/10.1145/2567574.2567620
Sao Pedro MGobert JBetts C(2014)Towards Scalable Assessment of Performance-Based Skills12th International Conference on Intelligent Tutoring Systems - Volume 847410.1007/978-3-319-07221-0_75(591-600)Online publication date: 5-Jun-2014
https://dl.acm.org/doi/10.1007/978-3-319-07221-0_75

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Examining change in students’ self-regulated learning patterns after a formative assessment using process mining techniques

On the role of gamification and localization in an open online learning environment: javala experiences

A supervised learning framework for learning management systems

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations