Abstract
There are well-acknowledged challenges to scaling computerized performance-based assessments. One such challenge is reliably and validly identifying ill-defined skills. We describe an approach that leverages a data mining framework to build and validate a detector that evaluates an ill-defined inquiry process skill, designing controlled experiments. The detector was originally built and validated for use with physical science simulations that have a simpler, linear causal structure. In this paper, we show that the detector can be used to identify demonstration of skill within a life science simulation on Ecosystems that has a complex underlying causal structure. The detector is evaluated in three ways: 1) identifying skill demonstration for a new student cohort, 2) handling the variability in how students conduct experiments, and 3) using it to determine when students are off-track before they finish collecting data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Shute, V.: Stealth Assessment in Computer-Based Games to Support Learning. In: Computer Games and Instruction, Charlotte, NC, pp. 503–523. Information Age Publishing (2011)
Clarke-Midura, J., Dede, C., Norton, J.: The Road Ahead for State Assessments, Cambridge, MA. Policy Analysis for California Education and Rennie Center for Educational Research & Policy (2011)
Gobert, J., Sao Pedro, M., Baker, R., Toto, E., Montalvo, O.: Leveraging educational data mining for real time performance assessment of scientific inquiry skills within microworlds. Journal of Educational Data Mining 4(1), 111–143 (2012)
Rupp, A.A., Gushta, M., Mislevy, R.J., Shaffer, D.W.: Evidence-centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments. The Journal of Technology, Learning, and Assessment 8(4), 1–45 (2010)
Shute, V., Glaser, R., Raghavan, K.: Inference and Discovery in an Exploratory Laboratory. In: Learning and Individual Differences: Advances in Theory and Research, pp. 279–326. W.H. Freeman, New York (1989)
Sao Pedro, M.A., Baker, R.S.J.D., Gobert, J.D., Montalvo, O., Nakama, A.: Leveraging Machine-Learned Detectors of Systematic Inquiry Behavior to Estimate and Predict Transfer of Inquiry Skil. User Modeling and User-Adapted Interaction 23, 1–39 (2013)
Sao Pedro, M., Baker, R., Gobert, J.: Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. In: Proc. of the 20th Conf. on User Modeling, Adaptation, and Personalization, Montreal, QC, Canada, pp. 249–260 (2012)
Sao Pedro, M.A., Baker, R.S.J.D., Gobert, J.D.: What Different Kinds of Stratification Can Reveal about the Generalizability of Data-Mined Skill Assessment Models. In: Proc. of the 3rd Conference on Learning Analytics and Knowledge, Leuven, Belgium (2013)
Gobert, J., Sao Pedro, M., Raziuddin, J., Baker, R.: From Log Files to Assessment Metrics for Science Inquiry using Educational Data Mining. Journal of the Learning Sciences 22(4), 521–563 (2013)
Greiff, S., Wustenberg, S., Funke, J.: Dynamic Problem Solving: A New Measurement Perspective. Applied Psychological Measurement 36, 189–213 (2012)
Chen, Z., Klahr, D.: All Other Things Being Equal: Acquisition and Transfer of the Control of Variables Strategy. Child Development 70(5), 1098–1120 (1999)
Sao Pedro, M.: Real-time Assessment, Prediction, and Scaffolding of Middle School Students’ Data Collection Skills within Physical Science Simulations. Ph.D. Dissertation etd-042513-062949, Worcester Polytechnic Institution, Worcester, MA (2013)
Yoon, S.: An Evolutionary Approach to Harnessing Complex Systems Thinking in the Science and Technology Classroom. Int’l Journal of Science Education 30(1), 1–32 (2008)
McElhaney, K., Linn, M.: Helping Students Make Controlled Experiments More Informative. In: Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences, Chicago, IL, pp. 786–793 (2010)
Baker, R. S. J. D., Corbett, A. T., Wagner, A. Z.: Human Classification of Low-Fidelity Replays of Student Actions. In : Proceedings of the Educational Data Mining Workshop held at the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Jhongli, Taiwan, pp.29-36 (2006)
Sao Pedro, M., Baker, R., Gobert, J.: Incorporating Scaffolding and Tutor Context into Bayesian Knowledge Tracing to Predict Inquiry Skill Acquisition. In: Proc. of the 6th International Conference on Educational Data Mining, Memphis, TN, pp. 185–192 (2013)
van Joolingen, W.R., de Jong, T.: An Extended Dual Search Space Model of Scientific Discovery Learning. Instructional Science 25, 307–346 (1997)
Hanley, J.A., McNeil, B.J.: The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143, 29–36 (1982)
Ben-David, A.: About the Relationship between ROC Curves and Cohen’s Kappa. Engineering Applications of Artificial Intelligence 21, 874–882 (2008)
Baker, R.S.J.D., Corbett, A.T., Roll, I., Koedinger, K.R.: Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction 18(3), 287–314 (2008)
Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., Heffernan, C.: Population Validity for Educational Data Mining Models: A Case Study in Affect Detection. To appear in the British Journal of Educational Technology (accepted)
San Pedro, M.O.C.Z., Baker, R.S.J.D., Rodrigo, M. M.T.: Detecting Carelessness through Contextual Estimation of Slip Probabilities among Students Using an Intelligent Tutor for Mathematics. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS (LNAI), vol. 6738, pp. 304–311. Springer, Heidelberg (2011)
Liu, O., Lee, H., Linn, M.C.: Multifaceted Assessment of Inquiry-Based Science Learning, pp. 69–86 (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Sao Pedro, M.A., Gobert, J.D., Betts, C.G. (2014). Towards Scalable Assessment of Performance-Based Skills: Generalizing a Detector of Systematic Science Inquiry to a Simulation with a Complex Structure. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds) Intelligent Tutoring Systems. ITS 2014. Lecture Notes in Computer Science, vol 8474. Springer, Cham. https://doi.org/10.1007/978-3-319-07221-0_75
Download citation
DOI: https://doi.org/10.1007/978-3-319-07221-0_75
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07220-3
Online ISBN: 978-3-319-07221-0
eBook Packages: Computer ScienceComputer Science (R0)