skip to main content
10.1145/3027385.3027445acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article
Public Access

What'd you say again?: recurrence quantification analysis as a method for analyzing the dynamics of discourse in a reading strategy tutor

Published: 13 March 2017 Publication History

Abstract

In this study, we investigated the degree to which the cognitive processes in which students engage during reading comprehension could be examined through dynamical analyses of their natural language responses to texts. High school students (n = 142) generated typed self-explanations while reading a science text. They then completed a comprehension test that measured their comprehension at both surface and deep levels. The recurrent patterns of the words in students' self-explanations were first visualized in recurrence plots. These visualizations allowed us to qualitatively analyze the different self-explanation processes of skilled and less skilled readers. These recurrence plots then allowed us to calculate recurrence indices, which represented the properties of these temporal word patterns. Results of correlation and regression analyses revealed that these recurrence indices were significantly related to the students' comprehension scores at both surface- and deep levels. Additionally, when combined with summative metrics of word use, these indices were able to account for 32% of the variance in students' overall text comprehension scores. Overall, our results suggest that recurrence quantification analysis can be utilized to guide both qualitative and quantitative assessments of students' comprehension.

References

[1]
Allen, L. K., Jacovina, M. E., and McNamara, D. S. 2016. Cohesive features of deep text comprehension processes. In J. Trueswell, A. Papafragou, D. Grodner, and D. Mirman (Eds.), Proceedings of the 38th Annual Meeting of the Cognitive Science Society in Philadelphia, PA, 2681--2686. Austin, TX: Cognitive Science Society.
[2]
Allen, L. K., McNamara, D. S., and McCrudden, M. T. 2015. Change your mind: Investigating the effects of self-explanation in the resolution of misconceptions. In D. C. Noelle, R. Dale, A. S. Warlaumont, J. Yoshimi, T. Matlock, C. D. Jennings, and P. Maglio, (Eds.), Proceedings of the 37th Annual Meeting of the Cognitive Science Society (Cog Sci 2015), 78--83. Pasadena, CA: Cognitive Science Society.
[3]
Allen, L. K., Snow, E. L., and McNamara, D. S. 2015. Are you reading my mind? Modeling students' reading comprehension skills with Natural Language Processing techniques. In J. Baron, G. Lynch, N. Maziarz, P. Blikstein, A. Merceron, and G. Siemens (Eds.), Proceedings of the 5th International Learning Analytics & Knowledge Conference (LAK'15), 246--254. Poughkeepsie, NY: ACM.
[4]
Anderson, N. C., Bischof, W. F., Laidlaw, K. E., Risko, E. F. and Kingstone, A. 2013. Recurrence quantification analysis of eye movements. Behavior research methods, 45(3), 842--856.
[5]
Baker, R., and Ocumpaugh, J. 2015. Interaction-based affect detection in educational software. In R. Calvo, S. D'Mello, J. Gratch & A. Kappas (Eds.), The Oxford handbook of affective computing, 233--245. New York: Oxford University Press.
[6]
Boekaerts, M., 2002. The on-line motivation questionnaire: A self-report instrument to assess students' context sensitivity. Advances in motivation and achievement, 12, 77--120.
[7]
Brusilovsky, P. 1994. The construction and application of student models in intelligent tutoring systems. Journal of Computer and Systems Science International, 23, 70--89.
[8]
Chi, M., Bassok, M., Lewis, M., Reimann, P., and Glaser, R. 1989. Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145--182.
[9]
Coco, M. I. and Dale, R. 2013. Cross-recurrence quantification analysis of categorical and continuous time series: an R package. arXiv preprint arXiv:1310.0201.
[10]
Crossley, S. A. and McNamara, D. S. (Eds.). 2016. Adaptive educational technologies for literacy instruction. New York: Taylor & Francis, Routledge.
[11]
Dale, R. and Spivey, M. J., 2005. Categorical recurrence analysis of child language. In Proceedings of the 27th annual meeting of the cognitive science society, 530--535. Mahwah, NJ: Lawrence Erlbaum.
[12]
D'Mello, S., Dowell, N., and Graesser, A. 2009. Cohesion relationships in tutorial dialogue as predictors of affective states. In Dimitrova V., Mizoguchi R., du Boulay B., Graesser A. (eds.) Proceedings of the 14th International Conference on Artificial Intelligence in Education, 9--16. IOS Press, Amsterdam.
[13]
D'Mello, S. and Graesser, A. 2015. Feeling, thinking, and computing with affect-aware learning technologies. In R. Calvo, S. D'Mello, J. Gratch & A. Kappas (Eds.), The Oxford handbook of affective computing, 419--434. New York: Oxford University Press.
[14]
Duckworth, A. L., Peterson, C., Matthews, M. D., and Kelly, D. R. 2007. Grit: Perseverance and passion for long-term goals. Journal of personality and social psychology, 92(6), 1087--1101.
[15]
Frederick, S. 2005. Cognitive reflection and decision making. The Journal of Economic Perspectives, 19(4), 25--42.
[16]
Frishkoff, G. A., Collins-Thompson, K., Hodges, L., and Crossley, S., 2016. Accuracy feedback improves word learning from context: evidence from a meaning-generation task. Reading and Writing, 29(4), 609--632.
[17]
Geiser, S. and Studley, R. 2001. UC and SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Oakland, CA: University of California.
[18]
Graesser, A. C., Chipman, P., King, B., McDaniel, B., and D'Mello, S. 2007. Emotions and learning with auto tutor. Frontiers in Artificial Intelligence and Applications, 158, 569--571.
[19]
Graesser, A. C., Chipman, P., Haynes, B. C. and Olney, A. 2005. AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4), 612--618.
[20]
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., and Louwerse, M. M. 2004. AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2), 180--192.
[21]
Graesser, A. C., VanLehn, K., Rosé, C. P., Jordan, P. W. and Harter, D. 2001. Intelligent tutoring systems with conversational dialogue. AI magazine, 22(4), 39--51.
[22]
Guilford, J. P., Christensen, P. R., Merrifield, P. R., and Wilson, R. C. 1978. Alternate uses: Manual of instructions and interpretation. Orange, CA: Sheridan Psychological Services.
[23]
Hannon, B. and Daneman, M. 2001. A new tool for measuring and understanding the individual differences in the component processes of reading comprehension. Journal of Educational Psychology, 93, 103--128.
[24]
Haswell, R. H. 2006. Automatons and automated scoring: Drudges, black boxes, and dei ex machina. In: P. F. Ericsson and R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences, 57--78. Logan, UT: Utah State University Press.
[25]
Jackson, G. T., Guess, R. H., and McNamara, D. S. 2010. Assessing cognitively complex strategy use in an untrained domain. Topics in Cognitive Science, 2, 127--137.
[26]
Jha, S. and Bhattacharyya, S. S. 2013. Learning orientation and performance orientation: scale development and its relationship with performance. Global Business Review, 14(1), 43--54.
[27]
MacGinitie, W. H. and MacGinitie, R. K. 1989. Gates MacGinitie reading tests. Chicago, IL: Riverside.
[28]
Magliano, J., Todar, S., Millis, K., Wiemer-Hastings, K., Kim, H., and McNamara, D. 2005. Changes in reading strategies as a function of reading training: A comparison of live and computerized training. Journal of Educational Computing Research, 32, 185--208.
[29]
Marwan, N., Romano, M. C., Thiel, M., and Kurths, J., 2007. Recurrence plots for the analysis of complex systems. Physics reports, 438(5), 237--329.
[30]
Marwan, N., Wessel, N., Meyerfeldt, U., Schirdewan, A., and Kurths, J. 2002. Recurrence-plot-based measures of complexity and their application to heart-rate-variability data. Physical review E, 66(2), 1--8.
[31]
McNamara, D. S. 2004. SERT: Self-explanation reading training. Discourse Processes, 38, 1--30.
[32]
McNamara, D. S. and Magliano, J. P. 2009. Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation. New York, NY: Elsevier Science.
[33]
McNamara, D. S., Levinstein, I. B., and Boonthum, C. 2004. iSTART: Interactive strategy trainer for active reading and thinking. Behavioral Research Methods, Instruments, & Computers, 36, 222--233.
[34]
McNamara, D. S., Boonthum, C., Levinstein, I. B., and Millis, K. 2007. Evaluating self-explanations in iSTART: Comparing word-based and LSA algorithms. In T. Landauer, D.S. McNamara, S. Dennis, and W. Kintsch (Eds.), Handbook of Latent Semantic Analysis, 227--241. Mahwah, NJ: Erlbaum.
[35]
Mednick, S., 1962. The associative basis of the creative process. Psychological review, 69(3), 220--232.
[36]
Murray, T. 1999. Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, 98--129.
[37]
National Assessment of Educational Progress. 2011. The nation's report card: Writing 2011. Retrieved Nov. 5, 2012, nces.ed.gov/nationsreportcard/writing.
[38]
Nkambou, R., Mizoguchi, R., and Bourdeau, J. (Eds.). 2010. Advances in intelligent tutoring systems (Vol. 308). Springer Science & Business Media.
[39]
Pintrich, P. R. and De Groot, E. V. 1990. Motivational and self-regulated learning components of classroom academic performance. Journal of educational psychology, 82(1), 33--40.
[40]
Powell, P. 2009. Retention and writing instruction: Implications for access and pedagogy. College Composition and Communication, 66, 664--682.
[41]
Riley, M. A., Balasubramaniam, R., and Turvey, M. T., 1999. Recurrence quantification analysis of postural fluctuations. Gait & posture, 9(1), 65--78.
[42]
Rosé, C. P., Jordan, P., Ringenberg, M., Siler, S., VanLehn, K. and Weinstein, A. 2001. Interactive conceptual tutoring in Atlas-Andes. In Proceedings of AI in Education 2001 Conference, 151--153.
[43]
Shermis, M. and Burstein, J. (Eds.). 2003. Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Erlbaum.
[44]
Shockley, K., Santana, M. V., and Fowler, C. A., 2003. Mutual interpersonal postural constraints are involved in cooperative conversation. Journal of Experimental Psychology: Human Perception and Performance, 29(2), 326--332.
[45]
Shute, V. J. 2011. Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction, 503--524. Charlotte, NC: Information Age Publishers.
[46]
Shute, V. J. and Kim, Y. J. 2013. Formative and stealth assessment. In J. M. Spector, M. D. Merrill, J. Elen, and M. J. Bishop (Eds.), Handbook of Research on Educational Communications and Technology (4th Edition), 311--323. New York, NY: Lawrence Erlbaum Associates, Taylor & Francis Group.
[47]
VanLehn, K. 2006. The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16, 227--265.
[48]
VanLehn, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46, 197--221.
[49]
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., and Rose, C. P. 2007. When are tutorial dialogues more effective than training? Cognitive Science, 31, 3--62.
[50]
Varner, L. K., Jackson, G. T., Snow, E. L., and McNamara, D. S. 2013. Does size matter? Investigating user input at a larger bandwidth. In C. Boonthum-Denecke and G. M. Youngblood (Eds.), Proceedings of the 26th International Florida Artificial Intelligence Research Society (FLAIRS) Conference, 546--549. Menlo Park, CA: AAAI Press.
[51]
Warschauer, M., and Ware, P. 2006. Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10, 1--24.
[52]
Wijekumar, K. K., Harris, K. R., Graham, S., and Meyer, B. J. F. 2016. We-Write. In S. A. Crossley and D. S. McNamara (Eds.) Adaptive educational technologies for literacy instruction, 184--203. New York: Taylor & Francis, Routledge.
[53]
Zbilut, J. P. and Webber, C. L., 1992. Embeddings and delays as derived from quantification of recurrence plots. Physics letters A, 171, 3--4, 199--203.

Cited By

View all
  • (2024)High-performing Groups during Children's Collaborative Coding Activities: What Can Multimodal Data Tell Us?Proceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3655805(533-559)Online publication date: 17-Jun-2024
  • (2024)Synchrony Between Facial Expressions and Heart Rate Variability During Game-Based Learning: Insights from Cross-Wavelet TransformationTechnology Enhanced Learning for Inclusive and Equitable Quality Education10.1007/978-3-031-72315-5_7(90-104)Online publication date: 13-Sep-2024
  • (2023)Enhancing Self-Regulated Learning With Artificial Intelligence-Powered Learning AnalyticsEmergent Practices of Learning Analytics in K-12 Classrooms10.4018/979-8-3693-0066-4.ch004(57-83)Online publication date: 29-Dec-2023
  • Show More Cited By

Index Terms

  1. What'd you say again?: recurrence quantification analysis as a method for analyzing the dynamics of discourse in a reading strategy tutor

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference
        March 2017
        631 pages
        ISBN:9781450348706
        DOI:10.1145/3027385
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 13 March 2017

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. corpus linguistics
        2. dynamics
        3. intelligent tutoring systems
        4. natural language processing
        5. reading
        6. stealth assessment

        Qualifiers

        • Research-article

        Funding Sources

        • ONR
        • IES

        Conference

        LAK '17
        LAK '17: 7th International Learning Analytics and Knowledge Conference
        March 13 - 17, 2017
        British Columbia, Vancouver, Canada

        Acceptance Rates

        LAK '17 Paper Acceptance Rate 36 of 114 submissions, 32%;
        Overall Acceptance Rate 236 of 782 submissions, 30%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)92
        • Downloads (Last 6 weeks)11
        Reflects downloads up to 06 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)High-performing Groups during Children's Collaborative Coding Activities: What Can Multimodal Data Tell Us?Proceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3655805(533-559)Online publication date: 17-Jun-2024
        • (2024)Synchrony Between Facial Expressions and Heart Rate Variability During Game-Based Learning: Insights from Cross-Wavelet TransformationTechnology Enhanced Learning for Inclusive and Equitable Quality Education10.1007/978-3-031-72315-5_7(90-104)Online publication date: 13-Sep-2024
        • (2023)Enhancing Self-Regulated Learning With Artificial Intelligence-Powered Learning AnalyticsEmergent Practices of Learning Analytics in K-12 Classrooms10.4018/979-8-3693-0066-4.ch004(57-83)Online publication date: 29-Dec-2023
        • (2022)Understanding Distributed Tutorship in Online Language TutoringLAK22: 12th International Learning Analytics and Knowledge Conference10.1145/3506860.3506883(164-174)Online publication date: 21-Mar-2022
        • (2022)Artificial Intelligence Applications in K-12 Education: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2022.317935610(61905-61921)Online publication date: 2022
        • (2022)Exploration of the Impact of Interpersonal Communication and Coordination Dynamics on Team Effectiveness in Human-Machine TeamsInternational Journal of Human–Computer Interaction10.1080/10447318.2022.214300439:9(1841-1855)Online publication date: 11-Nov-2022
        • (2022)Order pattern recurrence for the analysis of complex systemsPhysica A: Statistical Mechanics and its Applications10.1016/j.physa.2022.128204607(128204)Online publication date: Dec-2022
        • (2022)Intermodality in Multimodal Learning Analytics for Cognitive Theory Development: A Case from Embodied Design for Mathematics LearningThe Multimodal Learning Analytics Handbook10.1007/978-3-031-08076-0_6(133-158)Online publication date: 9-Oct-2022
        • (2022)Analyzing Information-Gathering Behavioral Sequences During Game-Based Learning Using Auto-recurrence Quantification AnalysisLearning and Collaboration Technologies. Designing the Learner and Teacher Experience10.1007/978-3-031-05657-4_5(60-71)Online publication date: 16-Jun-2022
        • (2022)Exemplar Use-Cases for Training Teachers on Learning AnalyticsHandbook on Intelligent Techniques in the Educational Process10.1007/978-3-031-04662-9_6(103-119)Online publication date: 16-Jun-2022
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Login options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media