Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Intelligence in Education

AIED 2019: Artificial Intelligence in Education pp 345–357Cite as

Integrating Students’ Behavioral Signals and Academic Profiles in Early Warning System

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11625))

Abstract

In this paper, we investigated how students’ behavioral signals and incoming profiles can be integrated to describe and predict student success in a higher education’s STEM course. The results include three major findings. First, we found behavioral signals like the number of correct responses to in-class questions, the number of confusing slides, and the number of viewed slides and videos are stable predictors of student success across different periods of a semester. Second, from the mixed-effect modeling results, we could identify significant gender gaps between mid-level incoming GPA student groups. We also showed some possible course advising scenarios based on the interaction between student behaviors and incoming profile factors. Third, using both behavioral signals and incoming profiles, our weekly forecast model on student success achieved a 72% prediction accuracy. We believe these findings can set the stage for subsequent early warning system studies that use different types of student data. Further investigations on the causal relationships for suggested results and developing other novel predictive features for student success would be beneficial for designing a better early warning system.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    One of the authors, Dr. Samson, is a consultant to Echo360 Inc. and uses the Echo360 Active Learning Platform in his class.

  2. 2.

    The distribution of grades were: 178 students with A (90 or higher, 13.87%), 507 with B (80–90, 39.52%), 373 with C (70–80, 29.07%), 167 with D (60–70, 13.02%), and 58 with F (60 or lower, 4.52%).

  3. 3.

    We follow the definition of R1 university in here: https://en.wikipedia.org/wiki/List_of_research_universities_in_the_United_States.

  4. 4.

    The results for random intercepts were similar between GLMER_i and GLMER_s models.

  5. 5.

    Detailed prediction results can be found at http://bit.ly/nam-EWSpreds.

References

  1. Aguilar, S., Lonn, S., Teasley, S.D.: Perceptions and use of an early warning system during a higher education transition program. In: Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, pp. 113–117. ACM (2014)

    Google Scholar 

  2. Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-task behavior in the cognitive tutor classroom: when students game the system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 383–390. ACM (2004)

    Google Scholar 

  3. Bates, D., Máchler, M., Bolker, B., Walker, S.: Fitting linear mixed-effects models using lme4. J. Stat. Softwa. 67(1), 1–48 (2015). https://doi.org/10.18637/jss.v067.i01

    Article  Google Scholar 

  4. Beck, H.P., Davidson, W.D.: Establishing an early warning system: predicting low grades in college students from survey of academic orientations scores. Res. High. Educ. 42(6), 709–723 (2001)

    Article  Google Scholar 

  5. Bote-Lorenzo, M.L., Gómez-Sánchez, E.: Predicting the decrease of engagement indicators in a MOOC. In: Proceedings of the Seventh International Learning Analytics and Knowledge Conference, pp. 143–147. ACM (2017)

    Google Scholar 

  6. Brown, M.G., DeMonbrun, R.M., Teasley, S.D.: Don’t call it a comeback: academic recovery and the timing of educational technology adoption. In: Proceedings of the Seventh International Learning Analytics and Knowledge Conference, pp. 489–493. ACM (2017)

    Google Scholar 

  7. Campbell, J.P., DeBlois, P.B., Oblinger, D.G.: Academic analytics: a new tool for a new era. Educase Rev. 42(4), 40 (2007)

    Google Scholar 

  8. Chen, Y., Johri, A., Rangwala, H.: Running out of stem: a comparative study across stem majors of college students at-risk of dropping out early. In: Proceedings of the Eighth International Learning Analytics and Knowledge Conference, pp. 270–279. ACM (2018)

    Google Scholar 

  9. Davis, D., Jivet, I., Kizilcec, R.F., Chen, G., Hauff, C., Houben, G.J.: Follow the successful crowd: raising mooc completion rates through social comparison at scale. In: Proceedings of the Seventh International Learning Analytics and Knowledge Conference, pp. 454–463. ACM (2017)

    Google Scholar 

  10. Goldstein, P.J., Katz, R.N.: Academic analytics: the uses of management information and technology in higher education, vol. 8. Educause (2005)

    Google Scholar 

  11. Hlosta, M., Zdrahal, Z., Zendulka, J.: Ouroboros: early identification of at-risk students without models based on legacy data. In: Proceedings of the Seventh International Learning Analytics and Knowledge Conference, pp. 6–15. ACM (2017)

    Google Scholar 

  12. Hoffmann, B., Ritchie, D.: Using multimedia to overcome the problems with problem based learning. Instr. Sci. 25(2), 97–115 (1997)

    Article  Google Scholar 

  13. Kizilcec, R.F., Cohen, G.L.: Eight-minute self-regulation intervention raises educational attainment at scale in individualist but not collectivist cultures. Proc. Natl. Acad. Sci. U.S.A. 114(17), 4348–4353 (2017)

    Article  Google Scholar 

  14. Koester, B.P., Grom, G., McKay, T.A.: Patterns of gendered performance difference in introductory stem courses. arXiv preprint arXiv:1608.07565 (2016)

  15. Krumm, A.E., Waddington, R.J., Teasley, S.D., Lonn, S.: A learning management system-based early warning system for academic advising in undergraduate engineering. In: Larusson, J.A., White, B. (eds.) Learn. Anal., pp. 103–119. Springer, New York (2014). https://doi.org/10.1007/978-1-4614-3305-7_6

    Chapter  Google Scholar 

  16. Macfadyen, L.P., Dawson, S.: Mining LMS data to develop an “early warning system” for educators: a proof of concept. Comput. Educ. 54(2), 588–599 (2010)

    Article  Google Scholar 

  17. Nam, S., Lonn, S., Brown, T., Davis, C.S., Koch, D.: Customized course advising: Investigating engineering student success with incoming profiles and patterns of concurrent course enrollment. In: Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, pp. 16–25. ACM (2014)

    Google Scholar 

  18. Park, J., Yu, R., Rodriguez, F., Baker, R., Smyth, P., Warschauer, M.: Understanding student procrastination via mixture models. In: Proceedings of the International Conference on Educational Data Mining (2018)

    Google Scholar 

  19. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2018). https://www.R-project.org/

  20. Ramesh, A., Goldwasser, D., Huang, B., Daume III, H., Getoor, L.: Learning latent engagement patterns of students in online courses. In: Twenty-Eighth AAAI Conference on Artificial Intelligence (2014)

    Google Scholar 

  21. Sinha, T., Li, N., Jermann, P., Dillenbourg, P.: Capturing “attrition intensifying” structural traits from didactic interaction sequences of MOOC learners. In: Proceedings of the EMNLP 2014 Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses, pp. 42–49 (2014)

    Google Scholar 

  22. Skinner, E.A., Kindermann, T.A., Furrer, C.J.: A motivational perspective on engagement and disaffection: conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educ. Psychol. Meas. 69(3), 493–525 (2009)

    Article  MathSciNet  Google Scholar 

  23. Tomkins, S., Ramesh, A., Getoor, L.: Predicting post-test performance from online student behavior: a high school mooc case study. In: Proceedings of the International Conference on Educational Data Mining (2016)

    Google Scholar 

  24. Veeramachaneni, K., O’Reilly, U.M., Taylor, C.: Towards feature engineering at scale for data from massive open online courses. arXiv preprint arXiv:1407.5238 (2014)

  25. Waddington, R.J., Nam, S., Lonn, S., Teasley, S.D.: Improving early warning systems with categorized course resource usage. J. Learn. Anal. 3(3), 263–290 (2016)

    Article  Google Scholar 

  26. Waddington, R.J., Nam, S.: Practice exams make perfect: incorporating course resource use into an early warning system. In: Proceedings of the Fourth International Learning Analytics And Knowledge Conference, pp. 188–192. ACM (2014)

    Google Scholar 

  27. Wang, A.Y., Newlin, M.H.: Characteristics of students who enroll and succeed in psychology web-based classes. J. Educ. Psychol. 92(1), 137 (2000)

    Article  Google Scholar 

  28. Wang, A.Y., Newlin, M.H.: Predictors of performance in the virtual classroom: identifying and helping at-risk cyber-students. J. (Technol. Horiz. Educ.) 29(10), 21 (2002)

    Google Scholar 

  29. Yang, D., Kraut, R., Rose, C.P.: Exploring the effect of student confusion in massive open online courses. J. Educ. Data Min. 8(1), 52–83 (2016)

    Google Scholar 

  30. Zacharis, N.Z.: A multivariate approach to predicting student outcomes in web-enabled blended learning courses. Internet High. Educ. 27, 44–53 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Michigan, Ann Arbor, USA

    SungJin Nam & Perry Samson

Authors
  1. SungJin Nam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Perry Samson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to SungJin Nam .

Editor information

Editors and Affiliations

  1. University of Sao Paulo, Sao Paulo, Brazil

    Seiji Isotani

  2. University of Malaga, Málaga, Spain

    Eva Millán

  3. Carnegie Mellon University, Pittsburgh, PA, USA

    Amy Ogan

  4. DePaul University, Chicago, IL, USA

    Peter Hastings

  5. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce McLaren

  6. University College London, London, UK

    Rose Luckin

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nam, S., Samson, P. (2019). Integrating Students’ Behavioral Signals and Academic Profiles in Early Warning System. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science(), vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-23204-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-23203-0

  • Online ISBN: 978-3-030-23204-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation