skip to main content
10.1145/3025453.3025841acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

A Unified Framework for Knowledge Assessment and Progression Analysis and Design

Published: 02 May 2017 Publication History

Abstract

Designing engaging learning content is important but difficult, and typically involves a lot of manual specification. We present a unified framework that utilizes automatic problem decomposition and partial ordering graph construction to facilitate multiple workflows: knowledge assessment and progression analysis and design. We present results from a study with 847 participants in an online Japanese-language assessment tool demonstrating that our framework can efficiently measure student ability and predict student performance on specific problems. We also present results from analysis of curricula showing that the progressions of two different textbooks are surprisingly similar, and that our framework can lead to the discovery of general principles of expert progression design. Finally, we demonstrate automatic progression generation with desired sequencing and pacing, allowing for tailoring of progressions and mapping of parameters extracted from one curriculum onto another.

Supplementary Material

ZIP File (pn2929-file4.zip)
suppl.mov (pn2929p.mp4)
Supplemental video

References

[1]
Sami Abuhamdeh and Mihaly Csikszentmihalyi. 2012. The importance of challenge for the enjoyment of intrinsically motivated, goal-directed activities. Personality and Social Psychology Bulletin 38, 3 (2012), 317--330.
[2]
Dietrich Albert and Theo Held. 1994. Establishing knowledge spaces by systematical problem construction. In Knowledge structures. Springer, 81--115.
[3]
Vincent Aleven, Bruce M. McLaren, Jonathan Sewall, and Kenneth R. Koedinger. 2006. The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In International Conference on Intelligent Tutoring Systems. Springer, 61--70.
[4]
Vincent Aleven, Bruce M. Mclaren, Jonathan Sewall, and Kenneth R. Koedinger. 2009. A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education 19, 2 (2009), 105--154.
[5]
Erik Andersen. 2014. Automatic Scaffolding for Procedural Learning. Ph.D. Dissertation.
[6]
Erik Andersen, Sumit Gulwani, and Zoran Popovic. 2013. A trace-based framework for analyzing and synthesizing educational progressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 773--782.
[7]
John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences 4, 2 (1995), 167--207.
[8]
John W. Atkinson, Jarvis R. Bastian, Robert W. Earl, and George H. Litwin. 1960. The achievement motive, goal setting, and probability preferences. The Journal of Abnormal and Social Psychology 60, 1 (1960), 27.
[9]
Eri Banno, Yoko Ikeda, and Yutaka Ohno. 2011. GENKI: An Integrated Course in Elementary Japanese. Japan Times and Tsai Fong Books.
[10]
Joseph E. Beck, Kai-min Chang, Jack Mostow, and Albert Corbett. 2008. Does help help? Introducing the Bayesian Evaluation and Assessment methodology. In International Conference on Intelligent Tutoring Systems. Springer, 383--394.
[11]
Eric Butler, Erik Andersen, Adam M. Smith, Sumit Gulwani, Zoran Popovic, and W. A. Redmond. 2015. Automatic Game Progression Design through Analysis of Solution Features. In Proc. of the SIGCHI Conf. on Human Factors in Computing (CHI 2015).
[12]
Eric Butler, Adam M. Smith, Yun-En Liu, and Zoran Popovic. 2013. A mixed-initiative tool for designing level progressions in games. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 377--386.
[13]
Herlinda Cancino, Ellen Rosansky, and John Schumann. 1978. The acquisition of English negatives and interrogatives by native Spanish speakers. Second language acquisition: A book of readings (1978), 207--230.
[14]
Albert T. Corbett and John R. Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253--278.
[15]
Mihaly Csikszentmihalyi. 1990. Flow: The Psychology of Optimal Experience. Harper & Row Publishers, Inc., New York, NY, USA. 49 pages.
[16]
Mihaly Csikszentmihalyi. 1991. Flow: The psychology of optimal experience. Vol. 41. HarperPerennial New York.
[17]
Gabriel Culbertson, Erik Andersen, Walker White, Daniel Zhang, and Malte Jung. 2016a. Crystallize: An Immersive, Collaborative Game for Second Language Learning. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 636--647.
[18]
Gabriel Culbertson, Shiyu Wang, Malte Jung, and Erik Andersen. 2016b. Social Situational Language Learning through an Online 3D Game. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 957--968.
[19]
Rafael Jaime De Ayala. 2013. The theory and practice of item response theory. Guilford Publications.
[20]
Edward L. Deci and Richard M. Ryan. 2000. The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry 11, 4 (2000), 227--268.
[21]
Michel C. Desmarais, Peyman Meshkinfam, and Michel Gagnon. 2006. Learned student models with item to item knowledge structures. User Modeling and User-Adapted Interaction 16, 5 (2006), 403--434.
[22]
Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O'Hara, and Dan Dixon. 2011. Gamification. using game-design elements in non-gaming contexts. In CHI'11 Extended Abstracts on Human Factors in Computing Systems. ACM, 2425--2428.
[23]
Jean-Paul Doignon and Jean-Claude Falmagne. 1985. Spaces for the assessment of knowledge. International journal of man-machine studies 23, 2 (1985), 175--196.
[24]
Jean-Paul Doignon and Jean-Claude Falmagne. 2012. Knowledge spaces. Springer Science & Business Media.
[25]
Tao Dong, Mira Dontcheva, Diana Joseph, Karrie Karahalios, Mark Newman, and Mark Ackerman. 2012. Discovery-based games for learning software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2083--2086.
[26]
Fritz Drasgow and Charles L Hulin. 1990. Item response theory. (1990).
[27]
Susan E Embretson and Steven P. Reise. 2013. Item response theory. Psychology Press.
[28]
Jean-Claude Falmagne, Eric Cosyn, Jean-Paul Doignon, and Nicolas Thiéry. 2006. The assessment of knowledge, in theory and in practice. In Formal concept analysis. Springer, 61--79.
[29]
Susan Gass. 1982. From Theory to Practice. (1982).
[30]
Jennifer M. Goldschneider and Robert M. DeKeyser. 2001. Explaining the "Natural Order of L2 Morpheme Acquisition" in English: A Meta-analysis of multiple determinants. Language learning 51, 1 (2001), 1--50.
[31]
Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics (TOG) 28, 3 (2009), 66.
[32]
Ronald K. Hambleton. 1991. Fundamentals of item response theory. Vol. 2. Sage publications.
[33]
Erik Harpstead, Brad A. Myers, and Vincent Aleven. 2013. In search of learning: facilitating data analysis in educational games. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 79--88.
[34]
Susan Harter. 1978. Effectance motivation reconsidered. Toward a developmental model. Human development 21, 1 (1978), 34--64.
[35]
Cord Hockemeyer, Theo Held, and Dietrich Albert. 1997. RATH-A relational adaptive tutoring hypertext WWW-environment based on knowledge space theory. (1997).
[36]
Andreas Holzer, Christian Schallhart, Michael Tautschnig, and Helmut Veith. 2008. Fshell: Systematic test case generation for dynamic analysis and measurement. In International Conference on Computer Aided Verification. Springer, 209--213.
[37]
Katherine Isbister, Mary Flanagan, and Chelsea Hash. 2010. Designing games for learning: insights from conversations with designers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2041--2044.
[38]
Edward L. Keenan and Bernard Comrie. 1977. Noun phrase accessibility and universal grammar. Linguistic inquiry 8, 1 (1977), 63--99.
[39]
Kenneth R. Koedinger, John R. Anderson, William H. Hadley, and Mary A. Mark. 1997. Intelligent tutoring goes to school in the big city. (1997).
[40]
Mathieu Koppen. 1993. Extracting human expertise for constructing knowledge spaces: An algorithm. Journal of mathematical psychology 37, 1 (1993), 1--20.
[41]
Mathieu Koppen and Jean-Paul Doignon. 1990. How to build a knowledge space by querying an expert. Journal of Mathematical Psychology 34, 3 (1990), 311--331.
[42]
Stephen D. Krashen. 1985. The input hypothesis: Issues and implications. Addison-Wesley Longman Ltd.
[43]
Nan Li, William W. Cohen, and Kenneth R. Koedinger. 2012. Problem order implications for learning transfer. In Intelligent Tutoring Systems. Springer, 185--194.
[44]
Conor Linehan, George Bellord, Ben Kirman, Zachary H. Morford, and Bryan Roche. 2014. Learning curves: analysing pace and challenge in four successful puzzle games. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play. ACM, 181--190.
[45]
Derek Lomas, Kishan Patel, Jodi L. Forlizzi, and Kenneth R. Koedinger. 2013. Optimizing challenge in an educational game using large-scale design experiments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 89--98.
[46]
Frederic M. Lord. 1980. Applications of item response theory to practical testing problems. Routledge.
[47]
Thomas W. Malone. 1980. What makes things fun to learn? Heuristics for designing instructional computer games. In Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems. ACM, 162--169.
[48]
Jürgen Meisel. 1987. Reference to past events and actions in the development of natural second language acquisition. First and second language acquisition processes (1987), 206--224.
[49]
Zachary Pardos and Neil Heffernan. 2010. Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm. In Educational Data Mining 2010.
[50]
Zachary A. Pardos and Neil T. Heffernan. 2011. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In International Conference on User Modeling, Adaptation, and Personalization. Springer, 243--254.
[51]
People's Education Press. 2013. Standard Japanese of China-Japan Exchanges for Beginners. Mitsumura Tosho Publishing Co.Ltd.
[52]
Manfred Pienemann, Malcolm Johnston, and Geoff Brindley. 1988. Constructing an acquisition-based procedure for second language assessment. Studies in second language acquisition 10, 02 (1988), 217--243.
[53]
Georg Rasch. 1993. Probabilistic models for some intelligence and attainment tests. ERIC.
[54]
Mark Reckase. 2009. Multidimensional item response theory. Vol. 150. Springer.
[55]
C. Reigeluth and R. Stein. 1983. Elaboration theory. Instructional-design theories and models: An overview of their current status (1983), 335--381.
[56]
Charles M. Reigeluth. 2013. Instructional design theories and models: An overview of their current status. Routledge.
[57]
Mare Taagepera and S. Noori. 2000. Mapping students' thinking patterns in learning organic chemistry by the use of knowledge space theory. J. Chem. Educ 77, 9 (2000), 1224.
[58]
Nikolai Tillmann and Jonathan De Halleux. 2008. Pex - white box test generation for. net. In International conference on tests and proofs. Springer, 134--153.
[59]
Lev Semenovich Vygotsky. 1980. Mind in society: The development of higher psychological processes. Harvard university press.
[60]
Shuhan Wang and Erik Andersen. 2016. Grammatical Templates: Improving Text Difficulty Evaluation for Language Learners. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics. 1692--1702.
[61]
Joanna White. 1998. Getting the learners' attention: A typographical input enhancement study. Focus on form in classroom second language acquisition (1998), 85--113.
[62]
Henning Wode. 1981. LANGUAGE-ACQUISITIONAL UNIVERSALS: A UNIFIED VIEW OF LANGUAGE ACQUISITION. Annals of the New York Academy of Sciences 379, 1 (1981), 218--234.
[63]
Ruth Wylie, Kenneth Koedinger, and Teruko Mitamura. 2010. Analogies, explanations, and practice: examining how task types affect second language grammar learning. In International Conference on Intelligent Tutoring Systems. Springer, 214--223.
[64]
Yanbo Xu and Jack Mostow. 2010. Using logistic regression to trace multiple sub-skills in a dynamic bayes net. In Educational Data Mining 2011.
[65]
Helen Zhao, Kenneth Koedinger, and John Kowalski. 2013. Knowledge tracing and cue contrast: Second language English grammar instruction. In Proceedings of the 35th Annual Conference of the Cognitive Science Society.
[66]
Helmut Zobl. 1984. The wave model of linguistic change and the naturalness of interlanguage. Studies in Second Language Acquisition 6, 02 (1984), 160--185.

Cited By

View all
  • (2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.235345141:2(1569-1584)Online publication date: 22-May-2024
  • (2022)Knowledge management of product designAdvanced Engineering Informatics10.1016/j.aei.2022.10154152:COnline publication date: 1-Apr-2022
  • (2021)How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?Proceedings of the ACM on Human-Computer Interaction10.1145/34746715:CHI PLAY(1-29)Online publication date: 6-Oct-2021
  • Show More Cited By

Index Terms

  1. A Unified Framework for Knowledge Assessment and Progression Analysis and Design

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. automatic problem decomposition
    2. education
    3. knowledge assessment
    4. progression analysis and design

    Qualifiers

    • Research-article

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.235345141:2(1569-1584)Online publication date: 22-May-2024
    • (2022)Knowledge management of product designAdvanced Engineering Informatics10.1016/j.aei.2022.10154152:COnline publication date: 1-Apr-2022
    • (2021)How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?Proceedings of the ACM on Human-Computer Interaction10.1145/34746715:CHI PLAY(1-29)Online publication date: 6-Oct-2021
    • (2021)Automatic Adaptive Sequencing in a WebgameIntelligent Tutoring Systems10.1007/978-3-030-80421-3_47(430-438)Online publication date: 7-Jun-2021
    • (2019)Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation GamesExtended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3341215.3356310(661-669)Online publication date: 17-Oct-2019
    • (2019)Goal-based progression synthesis in a korean learning gameProceedings of the 14th International Conference on the Foundations of Digital Games10.1145/3337722.3337745(1-9)Online publication date: 26-Aug-2019
    • (2019)PeerLensProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300864(1-12)Online publication date: 2-May-2019
    • (2019)Adaptive Learning Material Recommendation in Online Language EducationArtificial Intelligence in Education10.1007/978-3-030-23207-8_55(298-302)Online publication date: 21-Jun-2019
    • (2018)Combining adaptivity with progression ordering for intelligent tutoring systemsProceedings of the Fifth Annual ACM Conference on Learning at Scale10.1145/3231644.3231672(1-4)Online publication date: 26-Jun-2018
    • (2018)Course-Adaptive Content Recommender for Course AuthoringLifelong Technology-Enhanced Learning10.1007/978-3-319-98572-5_34(437-451)Online publication date: 14-Aug-2018
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media