research-article

Towards automatic experimentation of educational knowledge

Authors:

Emma Brunskill,

Zoran PopovićAuthors Info & Claims

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 3349 - 3358

https://doi.org/10.1145/2556288.2557392

Published: 26 April 2014 Publication History

Abstract

We present a general automatic experimentation and hypothesis generation framework that utilizes a large set of users to explore the effects of different parts of an intervention parameter space on any objective function. We also incorporate importance sampling, allowing us to run these automatic experiments even if we cannot give out the exact intervention distributions that we want. To show the utility of this framework, we present an implementation in the domain of fractions and numberlines, using an online educational game as the source of players. Our system is able to automatically explore the parameter space and generate hypotheses about what types of numberlines lead to maximal short-term transfer; testing on a separate dataset shows the most promising hypotheses are valid. We briefly discuss our results in the context of the wider educational literature, showing that one of our results is not explained by current research on multiple fraction representations, thus proving our ability to generate potentially interesting hypotheses to test.

Supplementary Material

MP4 File (p3349-sidebyside.mp4)

Download
184.50 MB

References

[1]

Ainsworth, S. E., Bibby, P. A., and Wood, D. J. Analysing the costs and benefits of multi-representational learning environments. Learning with multiple representations (1998), 120--134.

[2]

Andersen, E., Liu, Y.-E., Snider, R., Szeto, R., Cooper, S., and Popović, Z. On the harmfulness of secondary game objectives. In FDG (2011).

Digital Library

[3]

Andersen, E., Liu, Y.-E., Snider, R., Szeto, R., and Popović, Z. Placing a value on aesthetics in online casual games. In CHI (2011).

Digital Library

[4]

Andersen, E., O'Rourke, E., Liu, Y.-E., Snider, R., Lowdermilk, J., Truong, D., Cooper, S., and Popović, Z. The impact of tutorials on games of varying complexity. In CHI (2012).

Digital Library

[5]

Anderson, J. R., Corbett, A. T., Koedinger, K. R., and Pelletier, R. Cognitive tutors: Lessons learned. The Journal of the Learning Sciences 4, 2 (1995), 167--207.

[6]

Ansari, D. Effects of development and enculturation on number representation in the brain. Nature Reviews Neuroscience 9, 4 (2008), 278--291.

[7]

Arbreton, A. Student goal orientation and help-seeking strategy use.

[8]

Berman, F., Fox, G., and Hey, A. J. Grid computing: making the global infrastructure a reality, vol. 2. Wiley. com, 2003.

Digital Library

[9]

BrainPOP. http://www.brainpop.com/.

[10]

Buhrmester, M., Kwang, T., and Gosling, S. D. Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data Perspectives on Psychological Science 6, 1 (2011), 3--5.

[11]

Churchill, Gilbert A., J. A paradigm for developing better measures of marketing constructs. Journal of Marketing Research 16, 1 (1979), pp. 64--73.

[12]

Efron, B., and Tibshirani, R. An introduction to the bootstrap, vol. 57. CRC press, 1993.

[13]

Gagatsis, A., and Elia, I. The effects of different modes of representations on mathematical problem solving. In IGPME, vol. 2 (2004), 447--454.

[14]

Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 1 (1970), 97--109.

[15]

Henrich, J., Heine, S. J., and Norenzayan, A. The weirdest people in the world. Behavioral and Brain Sciences 33, 2--3 (2010), 61--83.

[16]

Hume, G., Michael, J., Rovick, A., and Evens, M. Hinting as a tactic in one-on-one tutoring. Journal of the Learning Sciences 5, 1 (1996), 23--49.

[17]

King, R. D., Rowland, J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., Sparkes, A., Whelan, K. E., and Clare, A. The automation of science. Science 324, 5923 (2009), 85--89.

[18]

Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with mechanical turk. In CHI (2008).

Digital Library

[19]

Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Henne, R., Ferres, J. L., and Melamed, T. Online experimentation at microsoft. In Third Workshop on Data Mining Case Studies and Practice Prize (2009).

[20]

Kohavi, R., Henne, R. M., and Sommerfield, D. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In KDD (2007).

Digital Library

[21]

Langley, P. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987.

Digital Library

[22]

Larson, C. N. Locating proper fractions on number lines: Effect of length and equivalence. School Science and Mathematics 80, 5 (1980), 423--428.

[23]

Lee, Y., Buchanan, B., and Aronis, J. Knowledge-based learning in exploratory science: Learning rules to predict rodent carcinogenicity. Machine Learning 30, 2--3 (1998), 217--240.

Digital Library

[24]

Lesh, R., Post, T., and Behr, M. Representations and translations among representations in mathematics learning and problem solving. Problems of representation in the teaching and learning of mathematics (1987), 33--40.

[25]

Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, ACM (New York, NY, USA, 2010), 661--670.

Digital Library

[26]

Lomas, D., Patel, K., Forlizzi, J. L., and Koedinger, K. R. Optimizing challenge in an educational game using large-scale design experiments. In CHI (2013).

Digital Library

[27]

Moss, J., and Case, R. Developing children's understanding of the rational numbers: A new model and an experimental curriculum. Journal for Research in Mathematics Education 30, 2 (1999), pp. 122--147.

[28]

Quinlan, J. R. C4. 5: programs for machine learning, vol. 1. Morgan kaufmann, 1993.

Digital Library

[29]

Rau, M. A., Aleven, V., and Rummel, N. Intelligent tutoring systems with multiple representations and self-explanation prompts support learning of fractions. In AIED (2009).

Digital Library

[30]

Siegler, R. S., Thompson, C. A., and Schneider, M. An integrated theory of whole number and fractions development. Cognitive Psychology 62, 4.

[31]

Sierpinska, A. On understanding the notion of function. The concept of function: Aspects of epistemology and pedagogy 25 (1992), 23--58.

[32]

Singh, K., and Xie, M. Bootstrap: A statistical method, 2008.

[33]

Stamper, J. C., Lomas, D., Ching, D., Ritter, S., Koedinger, K. R., and Steinhart, J. The rise of the super experiment. In EDM (2012), 196--200.

Cited By

Bassen JBalaji BSchaarschmidt MThille CPainter JZimmaro DGames AFast EMitchell JBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Reinforcement Learning for the Adaptive Scheduling of Educational ActivitiesProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376518(1-12)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376518
Sarkar ACooper SArnedo JNacke LVanden Abeele VToups Dugas P(2019)Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation GamesExtended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3341215.3356310(661-669)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3341215.3356310
NeCamp TGardner JBrooks CHsiao SCunningham JMcCarthy KLynch GBrooks CFerguson RHoppe U(2019)Beyond A/B TestingProceedings of the 9th International Conference on Learning Analytics & Knowledge10.1145/3303772.3303812(539-548)Online publication date: 4-Mar-2019
https://dl.acm.org/doi/10.1145/3303772.3303812
Show More Cited By

Index Terms

Towards automatic experimentation of educational knowledge
1. Human-centered computing

Recommendations

Optimizing challenge in an educational game using large-scale design experiments
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Online games can serve as research instruments to explore the effects of game design elements on motivation and learning. In our research, we manipulated the design of an online math game to investigate the effect of challenge on player motivation and ...
Towards understanding balancing in exertion games
ACE '14: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology

Playing exertion games with others can be engaging. However, players with different physical skill levels competing against each other can experience reduced engagement because they are either not challenged enough, or challenged too much. Balancing ...
Towards automatic functional test execution
IUI '12: Proceedings of the 2012 ACM international conference on Intelligent User Interfaces

As applications are developed, functional tests ensure they continue to function as expected. Nowadays, functional testing is mostly done manually, with human testers verifying a system's functionality themselves, following hand-written instructions. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2014

4206 pages

ISBN:9781450324731

DOI:10.1145/2556288

General Chairs:
Matt Jones
Swansea University, Wales, UK
,
Philippe Palanque
Université Paul Sabatier, France
,
Program Chairs:
Albrecht Schmidt
University of Stuttgart, Germany
,
Tovi Grossman
Autodesk Research, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '14

Sponsor:

SIGCHI

CHI '14: CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2014

Ontario, Toronto, Canada

Acceptance Rates

CHI '14 Paper Acceptance Rate 465 of 2,043 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
490
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bassen JBalaji BSchaarschmidt MThille CPainter JZimmaro DGames AFast EMitchell JBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Reinforcement Learning for the Adaptive Scheduling of Educational ActivitiesProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376518(1-12)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376518
Sarkar ACooper SArnedo JNacke LVanden Abeele VToups Dugas P(2019)Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation GamesExtended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3341215.3356310(661-669)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3341215.3356310
NeCamp TGardner JBrooks CHsiao SCunningham JMcCarthy KLynch GBrooks CFerguson RHoppe U(2019)Beyond A/B TestingProceedings of the 9th International Conference on Learning Analytics & Knowledge10.1145/3303772.3303812(539-548)Online publication date: 4-Mar-2019
https://dl.acm.org/doi/10.1145/3303772.3303812
Dudley JJacques JKristensson PBrewster SFitzpatrick GCox AKostakos V(2019)Crowdsourcing Interface Feature Design with Bayesian OptimizationProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300482(1-12)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300482
Heintz SLaw E(2018)Digital Educational GamesACM Transactions on Computer-Human Interaction10.1145/317788125:2(1-47)Online publication date: 11-Apr-2018
https://dl.acm.org/doi/10.1145/3177881
Papoušek JStanislav VPelánek RGašević DLynch GDawson SDrachsler HPenstein Rosé C(2016)Evaluation of an adaptive practice system for learning geography factsProceedings of the Sixth International Conference on Learning Analytics & Knowledge10.1145/2883851.2883884(134-142)Online publication date: 25-Apr-2016
https://dl.acm.org/doi/10.1145/2883851.2883884
Lomas JForlizzi JPoonwala NPatel NShodhan SPatel KKoedinger KBrunskill EKaye JDruin ALampe CMorris DHourcade J(2016)Interface Design Optimization as a Multi-Armed Bandit ProblemProceedings of the 2016 CHI Conference on Human Factors in Computing Systems10.1145/2858036.2858425(4142-4153)Online publication date: 7-May-2016
https://dl.acm.org/doi/10.1145/2858036.2858425
Khajah MRoads BLindsey RLiu YMozer MKaye JDruin ALampe CMorris DHourcade J(2016)Designing Engaging Games Using Bayesian OptimizationProceedings of the 2016 CHI Conference on Human Factors in Computing Systems10.1145/2858036.2858253(5571-5582)Online publication date: 7-May-2016
https://dl.acm.org/doi/10.1145/2858036.2858253
Papoušek JStanislav VPelánek R(2016)Impact of Question Difficulty on Engagement and LearningProceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 968410.1007/978-3-319-39583-8_28(267-272)Online publication date: 7-Jun-2016
https://dl.acm.org/doi/10.1007/978-3-319-39583-8_28
Liu YBallweber CO'rourke EButler EThummaphan PPopović Z(2015)Large-Scale Educational CampaignsACM Transactions on Computer-Human Interaction10.1145/269976022:2(1-24)Online publication date: 10-Mar-2015
https://dl.acm.org/doi/10.1145/2699760
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten