Stephen E. Fancsali
ABSTRACT
If we wish to embed assessment for accountability within instruction, we need to better understand the relative contribution of different types of learner data to statistical models that predict scores on assessments used for accountability purposes. The present work scales up and extends predictive models of math test scores from existing literature and specifies six categories of models that incorporate information about student prior knowledge, socio-demographics, and performance within the MATHia intelligent tutoring system. Linear regression and random forest models are learned within each category and generalized over a sample of 23,000+ learners in Grades 6, 7, and 8 over three academic years in Miami-Dade County Public Schools. After briefly exploring hierarchical models of this data, we discuss a variety of technical and practical applications, limitations, and open questions related to this work, especially concerning to the potential use of instructional platforms like MATHia as a replacement for time-consuming standardized tests.
- N.O. Anozie and B.W. Junker 2006. Predicting end-of-year accountability assessment scores from monthly student records in an online tutoring system. American Association for Artificial Intelligence Workshop on Educational Data Mining (AAAI-06), July 17, 2006, Boston, MA.Google Scholar
- E. Ayers and B.W. Junker. 2008. IRT Modeling of Tutor Performance to Predict End-of-Year Exam Scores. Educational and Psychological Measurement, 68(6), 972--987.Google ScholarCross Ref
- R.S.J.d. Baker, A.T. Corbett, I. Roll, K.R. Koedinger. 2008. Developing a generalizable detector of when students game the system. User Model. User-Adap, 18, 287--314. Google ScholarDigital Library
- R.S.J.d. Baker, S.M. Gowda, M. Wixon, J. Kalka, A.Z. Wagner, A. Salvi,, V. Aleven, G.W. Kusbit, J, Ocumpaugh, L. Rossi. 2012. Towards sensor-free affect detection in Cognitive Tutor Algebra. In Proceedings of the 5th International Conference on Educational Data Mining, K. Yacef, O. Zaiane, A. Hershkovitz, M. Yudelson, J. Stamper (Eds.). 126--133.Google Scholar
- J.E. Beck, P. Lia, and J, Mostow. 2004. Automatically assessing oral reading fluency in a tutor that listens. Technology, Instruction, Cognition and Learning, 2(1-2), 61--81.Google Scholar
- A. Binet. 1909. Les idées modernes sur les enfants. {Modern concepts concerning children.} Flammarion: Paris.Google Scholar
- B.S. Bloom. 1968. Learning for Mastery. Evaluation comment, 1(2). UCLA Center for the Study of Evaluation of Instructional Programs, Los Angeles.Google Scholar
- L. Breiman. 1996. Bagging predictors. Machine Learning, 24(2), 123--140. Google ScholarCross Ref
- L. Breiman. 2001. Random forests. Machine Learning, 45(1), 5--32. Google ScholarDigital Library
- B.R. Buckingham. 1921. Intelligence and its measurement: A symposium. Journal of Educational Psychology, 12, 271--275.Google ScholarCross Ref
- J.C. Campione and A.L. Brown. 1985. Dynamic Assessment: One Approach and Some Initial Data. Technical Report No. 361.Champaign, IL: University of Illinois at Urbana-Champaign, Center for the Student of Reading.Google Scholar
- A.T. Corbett, J.R. Anderson. 1995. Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Model. User-Adap., 4, 253--278.Google ScholarCross Ref
- D. Danks and A.J. London. 2017. Algorithmic bias in autonomous systems. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, C. Sierra (Ed.). 4691--4697. International Joint Conferences on Artificial Intelligence. Google ScholarDigital Library
- M. Feng, N.T. Heffernan, and K.R. Koedinger. 2006. Predicting state test scores better with intelligent tutoring systems: developing metrocs to measure assistance required. In Proceedings of the Eighth International Conference on Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and T-W. Chan (Eds.). Springer-Verlag: Berlin, 31--40. Google ScholarDigital Library
- Florida Department of Education. 2014. FCAT 2.0 and Florida EOC Assessments Achievement Levels. Florida Department of Education: Tallahassee, FL. Retrieved 29 September 2017. http://www.fldoe.org/core/fileparse.php/3/urlt/achlevel.pdfGoogle Scholar
- Florida Department of Education. 2017. Florida Standards Assessment: 2016--17 FSA English Language Arts and Mathematics Fact Sheet. Retrieved 29 September 2017. http://www.fldoe.org/core/fileparse.php/5663/urlt/ELA-MathFSAFS1617.pdfGoogle Scholar
- E.L. Grigorenko and R.J. Sternberg. 1998. Dynamic Testing. Psychol. Bull. 124, 1 (1998), 75--111.Google ScholarCross Ref
- R. Hart, M. Casserly, R. Uzzell, M. Palacios, A. Corcoran, and A. Spurgeon. (2015). Student Testing in America's Great City Schools: An Inventory and Preliminary Analysis. Council of Great City Schools, Washington, DC.Google Scholar
- A. Joshi, S.E. Fancsali, S. Ritter, T. Nixon. 2014. Generalizing and Extending a Predictive Model for Standardized Test Scores Based On Cognitive Tutor Interactions. In Proceedings of the 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, B.M. McLaren (Eds.). 369--370.Google Scholar
- B.W. Junker. 2006. Using on-line tutoring records to predict end-of-year exam scores: experience with the ASSISTments project and MCAS 8th grade mathematics. In Assessing and modeling cognitive development in school: intellectual growth and standard settings, R.W. Lissitz (Ed.). Maple Grove, MN: JAM.Google Scholar
- K.R. Koedinger, A.T. Corbett, C. Perfetti. 2012. The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36 (5), 757--798.Google ScholarCross Ref
- S. Moses. 2017. State testing starts today; Opt Out CNY leader says changes are `smoke and mirrors.' Syracuse.com. Retrieved 29 September 2017. http://www.syracuse.com/schools/index.ssf/2017/03/opt-out_movement_ny_teacher_union_supports_parents_right_to_refuse_state_tests.htmlGoogle Scholar
- C. O'Neil. 2016. Weapons of math destruction: How big data increases inequality and threatens democracy. New York: Crown Publishers. Google Scholar
- J.F. Pane, B.A. Griffin, D.F. McCaffrey, R. Karam. 2014. Effectiveness of Cognitive Tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127--144.Google ScholarCross Ref
- Z.A. Pardos, R.S.J.d. Baker, M.O.C.Z. San Pedro, S,M. Gowda, S.M. Gowda. 2014. Affective States and State Tests: Investigating How Affect and Engagement during the School Year Predict End-of-Year Learning Outcomes. Journal of Learning Analytics, 1(1), 107--128.Google ScholarCross Ref
- Z.A. Pardos, N.T. Heffernan, B. Anderson, C. Heffernan. 2010. Using Fine Grained Skill Models to Fit Student Performance with Bayesian Networks. In Handbook of Educational Data Mining, C. Romero, S. Ventura, S. R. Viola, M. Pechenizkiy and R.S.J. Baker (Eds.). CRC Press, Boca Raton, FL. 417--426.Google Scholar
- PDK/Gallup. 2015. 47th annual PDK/Gallup Poll of the Public's Attitudes Toward the Public Schools: Testing Doesn't Measure Up For Americans. Phi Delta Kappan, 97(1).Google Scholar
- L. Razzaq, et al. 2005. The Assistment Project: Blending Assessment and Assisting. In Proceedings of the 12th International Conference on Artificial Intelligence In Education, C-K. Looi, G.I. McCalla, B. Bredeweg, J. Breuker (Eds.). 555--562. Amsterdam: IOS. Google ScholarDigital Library
- G. Schwarz. 1978. Estimating the dimension of a model. Ann. Statist. 6(2), 461--464.Google ScholarCross Ref
- S. Ritter, J.R. Anderson, K.R. Koedinger, A.T. Corbett. 2007. Cognitive Tutor: applied research in mathematics education. Psychon. B. Rev., 14, 249--255.Google ScholarCross Ref
- S. Ritter, A. Joshi, S.E. Fancsali, T. Nixon. 2013. Predicting Standardized Test Scores from Cognitive Tutor Interactions.Google Scholar
- V.J. Shute and G.R. Moore. 2017. Consistency and Validity in Game-Based Stealth Assessment. In Technology Enhanced Innovative Assessment: Development, Modeling, and Scoring From an Interdisciplinary Perspective, H. Jiao and R.W. Lissitz (Eds.). Information Age Publishing, Charlotte, NC, 31--51.Google Scholar
- R.E. Snow and D.F. Lohman. 1989. Implications of cognitive psychology for educational measurement. In Educational Measurement, R.L. Linn (Ed.). 3rd Edition. 263--331. New York: American Council on Education/Macmillan.Google Scholar
- State of Minnesota, Office of the Legislative Auditor. 2017. Standardized student testing: 2017 evaluation report. Retrieved 29 September 2017. http://www.auditor.leg.state.mn.us/ped/pedrep/studenttesting.pdfGoogle Scholar
- D. Wiliam. 2011. Embedded Formative Assessment. Bloomington, IN: Solution Tree Press.Google Scholar
Index Terms
- Using embedded formative assessment to predict state summative test scores
Recommendations
Online formative assessment in higher education: A review of the literature
As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non traditional spaces. These issues include concepts such as ...
Formative assessment in b-learning: effectively monitoring students learning
TEEM '14: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing MulticulturalityOne of the objectives of Higher Education is the development of learner autonomy so that students can become effective and efficient learners, with the capability for lifelong learning and for actively engage in the development of a knowledge society. ...
Implementation of a Mathematics Formative Assessment Online Tool Before and During Remote Learning
Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral ConsortiumAbstractASSISTments is a free online learning tool for improving students’ mathematics achievement by providing immediate feedback and hints to students, detailed information on how students performed to teachers, and instructional suggestions for ...
Comments