Skip to main content
SpringerLink
Log in

How do A/B Testing and Secondary Data Analysis on AIED Systems Influence Future Research?

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13355))

Included in the following conference series:

  • 3749 Accesses

Abstract

Recent years have seen a surge in research conducted on intelligent online learning platforms, with a particular expansion of research conducting A/B testing to decide which design to use, and research using secondary platform data in analyses. This scientometric study aims to investigate how scholarship builds on these two different types of research. We collected papers for both categories - A/B testing, and educational data mining (EDM) on log data- in the context of the same learning platform. We then collected a randomized stratified sample of papers citing those A/B and EDM papers, and coded the reason for each citation. On comparing the frequency of citation categories between the two types of papers, we found that A/B test papers were cited more often to provide background and context for a study, whereas the EDM papers were cited to use past specific core ideas, theories, and findings in the field. This paper establishes a method to compare the contribution of different types of research on AIED systems such as interactive learning platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The open released data sets are publicly available at https://www.etrialstestbed.org/resources/featured-studies/dataset-papers.

  2. 2.

    The data set created is publicly available at https://osf.io/rmswe/?view_only=d496417aef1e4046907d2271b8a86cbb.

References

  1. Baker, R.S., Yacef, K.: The state of educational data mining in 2009: a review and future visions. J. Educ. Data Mining 1(1), 3–17 (2009)

    Google Scholar 

  2. Beck, J.E., Arroyo, I., Woolf, B.P., Beal, C.” An ablative evaluation. In: Proceedings of the 9th International Conference on Artificial Intelligence in Education, pp. 611–613 (1999)

    Google Scholar 

  3. Bornmann, L., Daniel, H.: What do citation counts measure? A review of studies on citing behavior. J Document. 64, 45–80 (2009)

    Article  Google Scholar 

  4. Cambrosio, A., Limoges, C., Courtial, J., Laville, F.: Historical scientometrics? Mapping over 70 years of biological safety research with coword analysis. Scientometrics 27(2), 119–143 (1993)

    Article  Google Scholar 

  5. Chen, G., Rolim, V., Mello, R.F., Gašević, G.: Let's shine together! a comparative study between learning analytics and educational data mining. In: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pp. 544–553 (2020)

    Google Scholar 

  6. Cole, J.R., Cole, S.: The Ortega hypothesis: Citation analysis suggests that only a few scientists contribute to scientific progress. Science 178(4059), 368–375 (1972)

    Article  Google Scholar 

  7. Cronin, B.: The Citation Process: The Role and Significance of Citations in Scientific Communication, p. 103. Taylor Graham, London (1984)

    Google Scholar 

  8. Romero, C., Ventura, S.: Educational data mining and learning analytics: an updated survey. Wiley Interdiscipl. Rev. Data Mining Knowl. Discov. 10(3) (2020)

    Google Scholar 

  9. Peña-Ayala, A.: Educational data mining: a survey and a data mining-based analysis of recent works. Expert Syst. Appl. 41(4), 1432–1462 (2014)

    Article  Google Scholar 

  10. Dormezil, S., Khoshgoftaar, T., Robinson-Bryant, F.: Differentiating between educational data mining and learning analytics: a bibliometric approach. In: Proceedings of the Workshops of the International Conference on Educational Data Mining, pp. 17–22 (2019)

    Google Scholar 

  11. Fazeli, S., Drachsler, H., Sloep, P.: Socio-semantic networks of research publications in the learning analytics community. In: Proceedings of the LAK Data Challenge (2019)

    Google Scholar 

  12. Garfield, E.: Can citation indexing be automated. In: Symposium Proceedings of the Statistical Association Methods for Mechanized Documentation, pp. 189–192 (1965)

    Google Scholar 

  13. Garzone, M., Mercer, R.E.: Towards an automated citation classifier. In: Conference of the Canadian Society for Computational Studies of Intelligence, pp. 337–346 (2000)

    Google Scholar 

  14. Goel, G., Lallé, S., Luengo, V.: Fuzzy logic representation for student modelling. In: Proceedings of the International Conference on Intelligent Tutoring Systems, pp. 428–433 (2012)

    Google Scholar 

  15. Gross, P.L.K., Gross, E.M.K.: College libraries and chemical education. Science 66(1713), 385–389 (1927)

    Article  Google Scholar 

  16. Hopkins, A.L., Jawitz, J.W., McCarty, C., Goldman, A., Basu, N.: Disparities in publication patterns by gender, race and ethnicity based on a survey of a random sample of authors. Scientometrics 96(2), 515–534 (2013)

    Article  Google Scholar 

  17. Khajah, M., Lindsey, R.V., Mozer, M.C.: How deep is knowledge tracing? In: Proceedings of the International Conference on Educational Data Mining (2016)

    Google Scholar 

  18. Kizilcec, R., et al.: Scaling up behavioral science interventions in online education. Proc. Natl Acad. Sci. 117(26), 14900–14905 (2020)

    Article  Google Scholar 

  19. Koedinger, K.R., Baker, R.S., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J.: A data repository for the EDM community: the PSLC DataShop. In: Romero, C., Ventura, S., Pechenizkiy, M., Baker, Ryan S.J.d. (eds.) Handbook of Educational Data Mining, pp. 43–56. CRC Press, Boca Raton (2010)

    Google Scholar 

  20. Koedinger, K.R., Corbett, A.T., Perfetti, C.: The Knowledge-Learning-Instruction framework: bridging the science-practice chasm to enhance robust student learning. Cogn. Sci. 36(5), 757–798 (2012)

    Article  Google Scholar 

  21. Lindgren, L.: If Robert Merton said it, it must be true: A citation analysis in the field of performance measurement. Evaluation 17(1), 7–19 (2011)

    Article  Google Scholar 

  22. Liu, R., Koedinger, K.R.: Closing the Loop: Automated Data-Driven Cognitive Model Discoveries Lead to Improved Instruction and Learning Gains. Journal of Educational Data Mining 9(1), 25–41 (2017)

    Google Scholar 

  23. Malmi, L., Sheard, J., Kinnunen, P., Sinclair, S., Sinclair, J.: Theories and models of emotions, attitudes, and self-efficacy in the context of programming education. In: Proceedings of the 2020 ACM Conference on International Computing Education Research, pp. 36–47 (2020)

    Google Scholar 

  24. Maturana, R.A., Alvarado, M.E., López-Sola, S., Ibáñez, M.J., Elósegui, L.R.: Linked data based applications for learning analytics research: Faceted searches, enriched contexts, graph browsing and dynamic graphic visualisation of data. In: Proceedings of the LAK Data Challenge (2013)

    Google Scholar 

  25. Mostow, J., Beck, J.E., Valeri, J.: Can automated emotional scaffolding affect student persistence? A baseline experiment. In: Proceedings of the Workshop on “Assessing and Adapting to User Attitudes and Affect: Why, When and How?” at the 9th International Conference on User Modeling (UM'03), pp. 61–64 (2003)

    Google Scholar 

  26. Murphy, R., Roschelle, J., Feng, M., Mason, C.A.: Investigating efficacy, moderators and mediators for an online mathematics homework intervention. J. Res. Educ. Effect. 13(2), 235–270 (2020)

    Google Scholar 

  27. Ostrow, K., Heffernan, N., Williams, J.J.: Tomorrow’s edtech today: establishing a learning platform as a collaborative research tool for sound science. Teach. Coll. Rec. 119(3), 300–306 (2017)

    Article  Google Scholar 

  28. Paquette, L., Ocumpaugh, J., Li, Z., Andres, A., Baker, R.S.: Who’s Learning? Using Demographics in EDM Research. J. Educ. Data Min. 12(3), 1–30 (2020)

    Google Scholar 

  29. Park, J., Choi, H.J.: Factors influencing adult learners’ decision to drop out or persist in online learning. J. Educ. Technol. Soc. 12(4), 207–217 (2009)

    Google Scholar 

  30. Reich, J.: Rebooting MOOC research. Science 347(6217), 34–35 (2015)

    Article  Google Scholar 

  31. Shockley, W.: On the statistics of individual variations of productivity in research laboratories. Proc. IRE 45(3), 279–290 (1957)

    Article  Google Scholar 

  32. Stamper, J.C., et al.: The rise of the super experiment. In: Proceedings of the International Conference on Educational Data Mining Society (2012)

    Google Scholar 

  33. VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)

    Article  Google Scholar 

  34. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics dashboard applications. Am. Behav. Sci. 57(10), 1500–1509 (2013)

    Article  Google Scholar 

  35. Vinkler, P.: A quasi-quantitative citation model. Scientometrics 12(1–2), 47–72 (1987)

    Article  Google Scholar 

  36. Waheed, H., Hassan, S., Aljohani, N.R., Wasif, M.: A bibliometric perspective of learning analytics research landscape. Behav. Inf. Technol. 37, 10–11 (2018)

    Google Scholar 

  37. Wise, A.F., Jung, Y.: Teaching with analytics: Towards a situated model of instructional decision-making. J. Learn. Anal. 6(2), 53–69 (2019)

    Google Scholar 

  38. Yeung, C., Yeung, D.: Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In: Proceedings of ACM Conference on Learning at Scale, pp. 1–10 (2018)

    Google Scholar 

  39. Zhang, J., Shi, X., King, I., Yeung, D.: Dynamic key-value memory networks for knowledge tracing. In: Proceedings of the 26th International Conference on World Wide Web, pp. 765–774 (2017)

    Google Scholar 

  40. Zouaq, A., Joksimovic, S., Gasevic, D.: Ontology learning to analyze research trends in learning analytics publications. In: Proceedings of the LAK Data Challenge (2013)

    Google Scholar 

  41. Krichevsky, N., Spinelli, K., Heffernan, N., Ostrow, K., Emberling, M.R.: E-TRIALS, Doctoral dissertation, Worcester Polytechnic Institute (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Education, University of Pennsylvania, Philadelphia, USA

    Nidhi Nasiar, Ryan S. Baker, Jillian Li & Weiyi Gong

Authors
  1. Nidhi Nasiar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryan S. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jillian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiyi Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nidhi Nasiar .

Editor information

Editors and Affiliations

  1. Ateneo De Manila University, Quezon, Philippines

    Maria Mercedes Rodrigo

  2. Department of Computer Science, North Carolina State University, Raleigh, NC, USA

    Noburu Matsuda

  3. Durham University, Durham, UK

    Alexandra I. Cristea

  4. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nasiar, N., Baker, R.S., Li, J., Gong, W. (2022). How do A/B Testing and Secondary Data Analysis on AIED Systems Influence Future Research?. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11644-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11643-8

  • Online ISBN: 978-3-031-11644-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation