Skip to main content
Springer Nature Link
Log in

Implementing and Evaluating ASSISTments Online Math Homework Support At large Scale over Two Years: Findings and Lessons Learned

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13916))

Included in the following conference series:

  • 4920 Accesses

Abstract

Math performance continues to be an important focus for improvement. The most recent National Report Card in the U.S. suggested student math scores declined in the past two years possibly due to COVID-19 pandemic and related school closures. We report on the implementation of a math homework program that leverages AI-based one-to-one technology, in 32 schools for two years as a part of a randomized controlled trial in diverse settings of the state of North Carolina in the US. The program, called “ASSISTments,” provides feedback to students as they solve homework problems and automatically prepares reports for teachers about student performance on daily assignments. The paper describes the sample, the study design, the implementation of the intervention, including the recruitment effort, the training and support provided to teachers, and the approaches taken to assess teacher’s progress and improve implementation fidelity. Analysis of data collected during the study suggest that (a) treatment teachers changed their homework review practices as they used ASSISTments, and (b) the usage of ASSISTments was positively correlated with student learning outcome.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The study has been pre-registered on Registry of Efficacy and Effectiveness Studies (REES) https://sreereg.icpsr.umich.edu/framework/pdf/index.php?id=2064. The study has been approved by the Institutional Review Board at Worcester Polytechnic Institute and WestEd. Participating teachers all signed consent forms. Parents received a notification letter and opt-out form for their children.

  2. 2.

    Our agreement with NCERDC doesn’t permit sharing of obtained data with any third parties. Other data collected during the study has been deposited to the Open ICPSR data repository (https://www.openicpsr.org/openicpsr/project/183645/version/V1/view).

References

  1. Heffernan, N., Heffernan, C.: The ASSISTments ecosystem: building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. Int. J. Artif. Intell. Educ. 24(4), 470–497 (2014). https://doi.org/10.1007/s40593-014-0024-x

    Article  MathSciNet  Google Scholar 

  2. Roschelle, J., Feng, M., Murphy, R.F., Mason, C.A.: Online mathematics homework increases student achievement. AERA Open 2(4), 233285841667396 (2016). https://doi.org/10.1177/2332858416673968

    Article  Google Scholar 

  3. Murphy, R., Roschelle, J., Feng, M., Mason, C.: Investigating efficacy, moderators and mediators for an online mathematics homework intervention. J. Res. Educ. Effectiveness 13, 1–36 (2020). https://doi.org/10.1080/19345747.2019.1710885

    Article  Google Scholar 

  4. Sahni, S.D., et al.: A What Works Clearinghouse Rapid Evidence Review of Distance Learning Programs. U.S. Department of Education (2021)

    Google Scholar 

  5. Escueta, M., Quan, V., Nickow, A.J., Oreopoulos, P.: Education Technology: An Evidence-Based Review. NBER Working Paper No. 23744. August 2017 JEL No. I20,I29,J24 (2017). http://tiny.cc/NBER

  6. Pashler, H., Bain, P.M., Bottge, B.A., Graesser, A., Koedinger, K., McDaniel, M., et al.: Organizing instruction and study to improve student learning. IES practice guide (NCER 2007–2004). National Center for Education Research, Washington, D.C. (2007)

    Google Scholar 

  7. Heritage, M., Popham, W.J.: Formative Assessment in Practice: A Process of Inquiry and Action. Harvard Education Press, Cambridge, MA (2013)

    Google Scholar 

  8. Koedinger, K.R., Booth, J.L., Klahr, D.: Instructional complexity and the science to constrain it. Science 342(6161), 935–937 (2013). https://doi.org/10.1126/science.1238056

    Article  Google Scholar 

  9. Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77(1), 81–112 (2007)

    Article  Google Scholar 

  10. DiBattista, D., Gosse, L., Sinnige-Egger, J.A., Candale, B., Sargeson, K.: Grading scheme, test difficulty, and the immediate feedback assessment technique. J. Exp. Educ. 77(4), 311–338 (2009)

    Article  Google Scholar 

  11. Mendicino, M., Razzaq, L., Heffernan, N.T.: A comparison of traditional homework to computer supported homework. J. Res. Technol. Educ. 41(3), 331–358 (2009)

    Article  Google Scholar 

  12. Fyfe, E.R., Rittle-Johnson, B.: The benefits of computer-generated feedback for mathematics problem solving. J. Exp. Child Psychol. 147, 140–151 (2016)

    Article  Google Scholar 

  13. Kehrer, P., Kelly, K., Heffernan, N.: Does immediate feedback while doing homework improve learning? FLAIRS 2013. In: Proceedings of the 26th International Florida Artificial Intelligence Research Society Conference, pp. 542–545 (2013)

    Google Scholar 

  14. Kelly, K., Heffernan, N., Heffernan, C., Goldman, S., Pellegrino, J., Soffer-Goldstein, D.: Improving student learning in math through web-based homework review. In: Liljedahl, P., Nicol, C., Oesterle, S., Allan, D. (eds.) Proceedings of the Joint Meeting of PME 38 and PME-NA 36, vol. 3, pp. 417–424. PME, Vancouver, Canada (2014)

    Google Scholar 

  15. Black, P., Wiliam, D.: Classroom assessment and pedagogy. Assess. Educ.: Principles, Policy Pract. 25(6), 551–575 (2018)

    Article  Google Scholar 

  16. Bennett, R.E.: Formative assessment: a critical review. Assess. Educ.: Principles, Policy Pract. 18(1), 5–25 (2011)

    Article  Google Scholar 

  17. Dunn, K.E., Mulvenon, S.W.: A critical review of research on formative assessment: the limited scientific evidence of the impact of formative assessment in education. Pract. Assess. Res. Eval. 14(7), 1–11 (2009)

    Google Scholar 

  18. OECD: Formative Assessment: Improving Learning in Secondary Classrooms. OECD Publishing, Paris (2005). https://doi.org/10.1787/9789264007413-en

  19. Wiliam, D., Thompson, M.: Integrating assessment with learning: what will it take to make it work? In: Dwyer, C.A. (ed.) The Future of Assessment: Shaping Teaching and Learning, pp. 53–82. Lawrence Erlbaum Associates (2007)

    Google Scholar 

  20. Faber, J.M., Luyten, H., Visscher, A.J.: The effects of a digital formative assessment tool on mathematics achievement and student motivation: results of a randomized experiment. Comput. Educ. 106, 83–96 (2017)

    Article  Google Scholar 

  21. Koedinger, K.R., McLaughlin, E.A., Heffernan, N.T.: A quasi-experimental evaluation of an on-line formative assessment and tutoring system. J. Educ. Comput. Res. 43(4), 489–510 (2010). https://doi.org/10.2190/EC.43.4.d

    Article  Google Scholar 

  22. Sheard, M., Chambers, B., Elliott, L.: Effects of technology-enhanced formative assessment on achievement in primary grammar. Retrieved from the University of York website https://www.york.ac.uk/media/iee/documents/QfLGrammarReport_Sept2012.pdf (2012)

  23. Anderson, J.R.: Learning and Memory: An Integrated Approach, 2nd edn. John Wiley and Sons Inc., New York (2000)

    Google Scholar 

  24. Koedinger, K., Aleven, V.: Exploring the assistance dilemma in experiments with Cognitive Tutors. Educ. Psychol. Rev. 19, 239–264 (2007)

    Article  Google Scholar 

  25. Rohrer, D.: The effects of spacing and mixing practice problems. J. Res. Math. Educ. 40, 4–17 (2009)

    Article  Google Scholar 

  26. Rohrer, D., Taylor, K.: The shuffling of mathematics practice problems improves learning. Instr. Sci. 35, 481–498 (2007)

    Article  Google Scholar 

  27. Rohrer, D., Pashler, H.: Increasing retention without increasing study time. Curr. Dir. Psychol. Sci. 16, 183–186 (2007)

    Article  Google Scholar 

  28. Fairman, J., Porter, M., Fisher, S.: Principals Discuss Early Implementation of the ASSISTments Online Homework Tutor for Mathematics: ASSISTments Efficacy Study Report 2. SRI International, Menlo Park, CA (2015)

    Google Scholar 

  29. Loveless, T.: The 2014 Brown Center Report on American Education: How well are American students learning? In: The Brown Center on Education Policy. The Brookings Institution, Washington, DC (2014)

    Google Scholar 

  30. Bennett, S., Kalish, N.: The Case Against Homework: How Homework is Hurting Our Children and What We Can do About it. Crown Publishers, New York, NY (2006)

    Google Scholar 

  31. Kohn, A.: The Homework Myth: Why our Kids Get too Much of a Bad Thing. Da Capo Press, Cambridge, MA (2006)

    Google Scholar 

  32. Pope, D.: Making Homework Work. ASCD (2020). Retrieved 20 Oct 2021

    Google Scholar 

Download references

Acknowledgement

This material is based on work supported by the Institute of Education Sciences of the U.S. Department of Education under Grant R305A170641. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the funders.

Author information

Authors and Affiliations

  1. WestEd, San Francisco, California, 94107, USA

    Mingyu Feng & Kelly Collins

  2. Worcester Polytechnic Institute, Worcester, Massachusetts, 01609, USA

    Neil Heffernan

  3. The ASSISTments Foundation, Auburn, Massachusetts, 01501, USA

    Cristina Heffernan

  4. LFC Research, Mountwin view, California, 94043, USA

    Robert F. Murphy

Authors
  1. Mingyu Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neil Heffernan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kelly Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cristina Heffernan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robert F. Murphy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingyu Feng .

Editor information

Editors and Affiliations

  1. University of Southern California, Los Angeles, CA, USA

    Ning Wang

  2. University of British Columbia, Vancouver, BC, Canada

    Genaro Rebolledo-Mendez

  3. North Carolina State University, Raleigh, NC, USA

    Noboru Matsuda

  4. Despacho 3.01, UNED-Grupo de Investigación aDeNu, Madrid, Spain

    Olga C. Santos

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Feng, M., Heffernan, N., Collins, K., Heffernan, C., Murphy, R.F. (2023). Implementing and Evaluating ASSISTments Online Math Homework Support At large Scale over Two Years: Findings and Lessons Learned. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36272-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36271-2

  • Online ISBN: 978-3-031-36272-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics