skip to main content
10.1145/2724660.2724669acmconferencesArticle/Chapter ViewAbstractPublication Pagesl-at-sConference Proceedingsconference-collections
research-article

Addressing Common Analytic Challenges to Randomized Experiments in MOOCs: Attrition and Zero-Inflation

Published: 14 March 2015 Publication History

Abstract

Massive open online course (MOOC) platforms increasingly allow easily implemented randomized experiments. The heterogeneity of MOOC students, however, leads to two methodological obstacles in analyzing interventions to increase engagement. (1) Many MOOC participation metrics have distributions with substantial positive skew from highly active users as well as zero-inflation from high attrition. (2) High attrition means that in some experimental designs, most users assigned to the treatment never receive it; analyses that do not consider attrition result in "intent-to-treat" (ITT) estimates that underestimate the true effects of interventions. We address these challenges in analyzing an intervention to improve forum participation in the 2014 JusticeX course offered on the edX MOOC platform. We compare the results of four ITT models (OLS, logistic, quantile, and zero-inflated negative binomial regressions) and three "treatment-on-treated" (TOT) models (Wald estimator, 2SLS with a second stage logistic model, and instrumental variables quantile regression). A combination of logistic, quantile, and zero-inflated negative binomial regressions provide the most comprehensive description of the ITT effects. TOT methods then adjust the ITT underestimates. Substantively, we demonstrate that self-assessment questions about forum participation encourage more students to engage in forums and increases the participation of already active students.

References

[1]
Abadie, A., Angrist, J. and Imbens, G. Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica, 70, 1 ( 2002), 91--117. DOI=10.1111/1468-0262.00270.
[2]
Anderson, A., Huttenlocher, D., Kleinberg, J. and Leskovec, J. Engaging with Massive Online Courses. in Proceedings of the 2014 International World Wide Web Conference, (2014), 687--698.
[3]
Angrist, J. D. Mostly harmless econometrics: An empiricist's companion. Princeton: Princeton University Press, Princeton, 2009.
[4]
Bakshy, E., Eckles, D. and Bernstein, M. S. Designing and deploying online field experiments. in Proceedings of the 23rd international conference on World wide web. (2014), 283--292.
[5]
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D. and Seaton, D. Studying learning in the worldwide classroom: Research into edX's first MOOC. Research & Practice in Assessment, 8( 2013), 13--25.
[6]
Guo, P. J. and Reinecke, K. Demographic differences in how students navigate through MOOCs. in Proceedings of the First ACM Conference on Learning@Scale Conference. (Atlanta, USA, 2014). ACM, 21--30.
[7]
Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J. and Chuang, I. HarvardX and MITx: The First Year of Open Online Courses, Fall 2012-Summer 2013. HarvardX & MITx Working Paper No. 1., 2014.
[8]
Huang, J., Dasgupta, A., Ghosh, A., Manning, J. and Sanders, M. Superposter behavior in MOOC forums. in Proceedings of the First ACM Conference on Learning@Scale Conference. (Atlanta, USA, 2014). ACM, 117--126.
[9]
Kizilcec, R. F., Schneider, E., Cohen, G. and McFarland, D. Encouraging forum participation in online courses with collectivist, individualist, and neutral motivational framings. eLearning Papers, 37( 2014), 13--22.
[10]
Manning, J. and Sanders, M. How widely used are MOOC forums? A first look. Signals: Thoughts on Online Learning, ( 2013).
[11]
Nichols, A. IVPOIS: Stata module to estimate an instrumental variables Poisson regression via GMM. Statistical Software Components, ( 2008).
[12]
O'Hara, R. B. and Kotze, D. J. Do not log-transform count data. Methods in Ecology and Evolution, 1, 2 ( 2010), 118--122. DOI=10.1111/j.2041--210X.2010.00021.x.
[13]
Rabe-Hesketh, S., Skrondal, A. and Stata Corporation. Multilevel and longitudinal modeling using Stata. Stata Press Publication, College Station, Tex., 2008.
[14]
Reich, J. MOOC Completion and retention in the context of student intent. EDUCAUSE Review Online, (2014).
[15]
Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I. and Ho, A. D. JusticeX: Spring 2013 Course Report. HarvardX Working Paper No. 4., 2014.

Cited By

View all
  • (2024)More Likely to Pay but Less Engaged: The Effects of Switching Online Courses from Scheduled to On-Demand Release on User BehaviorJournal of Marketing10.1177/0022242924122714588:5(63-88)Online publication date: 20-Mar-2024
  • (2021)The “Idea Advantage”: How Content Sharing Strategies Impact Engagement in Online Learning PlatformsJournal of Marketing Research10.1177/0022243721101782859:1(61-78)Online publication date: 30-Sep-2021
  • (2021)Finding Place in a Design SpaceProceedings of the ACM on Human-Computer Interaction10.1145/34492465:CSCW1(1-30)Online publication date: 22-Apr-2021
  • Show More Cited By

Index Terms

  1. Addressing Common Analytic Challenges to Randomized Experiments in MOOCs: Attrition and Zero-Inflation

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      L@S '15: Proceedings of the Second (2015) ACM Conference on Learning @ Scale
      March 2015
      438 pages
      ISBN:9781450334112
      DOI:10.1145/2724660
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 14 March 2015

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. a/b testing
      2. attrition
      3. engagement
      4. moocs
      5. randomized controlled trials
      6. treatment-on-treated
      7. zero-inflation

      Qualifiers

      • Research-article

      Conference

      L@S 2015
      Sponsor:
      L@S 2015: Second (2015) ACM Conference on Learning @ Scale
      March 14 - 18, 2015
      BC, Vancouver, Canada

      Acceptance Rates

      L@S '15 Paper Acceptance Rate 23 of 90 submissions, 26%;
      Overall Acceptance Rate 117 of 440 submissions, 27%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)12
      • Downloads (Last 6 weeks)1
      Reflects downloads up to 12 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)More Likely to Pay but Less Engaged: The Effects of Switching Online Courses from Scheduled to On-Demand Release on User BehaviorJournal of Marketing10.1177/0022242924122714588:5(63-88)Online publication date: 20-Mar-2024
      • (2021)The “Idea Advantage”: How Content Sharing Strategies Impact Engagement in Online Learning PlatformsJournal of Marketing Research10.1177/0022243721101782859:1(61-78)Online publication date: 30-Sep-2021
      • (2021)Finding Place in a Design SpaceProceedings of the ACM on Human-Computer Interaction10.1145/34492465:CSCW1(1-30)Online publication date: 22-Apr-2021
      • (2021)How well do the students understand the course contents? Assessing comprehension through course videosJournal of Computers in Education10.1007/s40692-021-00205-99:3(315-349)Online publication date: 15-Oct-2021
      • (2021)Not Only the Intention to Complete: The Role of Action-Oriented Intentions in MOOC CompletionTechnology, Knowledge and Learning10.1007/s10758-021-09534-127:3(707-719)Online publication date: 26-Jun-2021
      • (2020)Upgrading Education with Technology: Insights from Experimental ResearchJournal of Economic Literature10.1257/jel.2019150758:4(897-996)Online publication date: 1-Dec-2020
      • (2020)Supporting learners' self-regulated learning in Massive Open Online CoursesComputers & Education10.1016/j.compedu.2019.103771146:COnline publication date: 1-Mar-2020
      • (2019)Refusing to TryProceedings of the 9th International Conference on Learning Analytics & Knowledge10.1145/3303772.3303806(391-400)Online publication date: 4-Mar-2019
      • (2019)Developing Early Detectors of Student Attrition and Wheel Spinning Using Deep LearningIEEE Transactions on Learning Technologies10.1109/TLT.2019.291216212:2(158-170)Online publication date: 1-Apr-2019
      • (2019)Systematic Review of Discussion Forums in Massive Open Online Courses (MOOCs)IEEE Transactions on Learning Technologies10.1109/TLT.2018.285930412:3(413-428)Online publication date: 1-Jul-2019
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media