Abstract
We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
National Commission on Writing: The Neglected “R”. College Entrance Examination Board, New York (2003)
Kellogg, R., Raulerson, B.: Improving the Writing Skills of College Students. Psychonomic Bulletin and Review 14, 237–242 (2007)
Graham, S., Perin, D.: A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology 99, 445–476 (2007)
McGarrell, H., Verbeem, J.: Motivating revision of drafts through formative feedback. ELT Journal 61, 228–236 (2007)
Devillez, R.: Writing: Step by Step. Kendall Hunt, Dubuque (2003)
Golightly, K., Sanders, G.: Writing and Reading in the Disciplines, vol. 2. Pearson, New Jersey (2000)
McNamara, D., Raine, R., Roscoe, R., Crossley, S., Jackson, G.T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Carney, D., Sullivan, S., Kim, L., Rus, V., Floyd, R., McCarthy, P., Graesser, A.: The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on Writing Strategies. In: McCarthy, P.M., Boonthum-Denecke, C. (eds.) Applied Natural Language Processing and Content Analysis: Identification, Investigation, and Resolution, pp. 298–311. IGI Global, Hershey (2012)
Graesser, A., McNamara, D., Louwerse, M., Cai, Z.: Coh-Metrix: Analysis of Text on Cohesion and Language. Behavioral Research Methods, Instruments and Computers 36, 193–202 (2004)
McNamara, D., Crossley, S., Roscoe, R.: Natural Language Processing in an Intelligent Writing Strategy Tutoring System. Behavioral Research Methods, Instruments and Computers (2012) (Advance online publication)
McNamara, D., Kintsch, E., Songer, N., Kintsch, W.: Are Good Texts Always Better? Interactions of Text Coherence, Background Knowledge, and Levels of Understanding in Learning from Text. Cognition and Instruction 14, 1–43 (1996)
Crossley, S., McNamara, D.: Cohesion, Coherence, and Expert Evaluations of Writing Proficiency. In: Ohlsson, S., Catrambone, R. (eds.) Proceedings of the 32nd Annual Conference of the Cognitive Science Society, pp. 984–989. Cognitive Science Society, Austin (2010)
Crossley, S., McNamara, D.: Text Coherence and Judgments of Essay Quality: Models of Quality and Coherence. In: Carlson, L., Hoelscher, C., Shipley, T.F. (eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society, pp. 1236–1241. Cognitive Science Society, Austin (2011)
Crossley, S., Weston, J., Sullivan, S., McNamara, D.: The Development of Writing Proficiency as a Function of Grade Level: A Linguistic Analysis. Written Communication 28, 282–311 (2011)
Crossley, S., McNamara, D.: Predicting Second Language Writing Proficiency: The Roles of Cohesion and Linguistic Sophistication. Journal of Research in Reading 53, 115–136 (2012)
Johnstone, K.M., Ashbaugh, H., Warfield, T.D.: Effects of repeated practice and contextual-writing experiences on college students’ writing skills. Journal of Educational Psychology 94, 305–315 (2002)
Kellogg, R.T., Raulerson, B.A.: Improving the writing skills of college students. Psychonomic Bulletin & Review 13(2), 237–242 (2007)
Graham, S., Harris, K.R.: The role of self-regulation and transcription skills in writing and writing development. Educational Psychologist 35, 3–12 (2000)
Grimes, D., Warschauer, W.: Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. Journal of Technology, Learning, and Assessment 8, 4–43 (2010)
Roscoe, R., Kugler, D., Crossley, S., Weston, J., McNamara, D.: Developing pedagogically-guided threshold algorithms for intelligent automated essay feedback. In: McCarthy, P., Youngblood, Y. (eds.) Proceedings of the 25th International Florida Artificial Intelligence Research Society (FLAIRS) Conference, pp. 466–471. The AAAI Press, Menlo Park (2012)
Crossley, S., Roscoe, R., McNamara, D.: Using Natural Language Processing Algorithms to Detect Changes in Student Writing in an Intelligent Tutoring System. Manuscript Submitted to the 26th International Florida Artificial Intelligence Research Society Conference (2013)
Hempelmann, C., Dufty, D., McCarthy, P., Graesser, A., Cai, Z., McNamara, D.: Using LSA to Automatically Identify Givenness and Newness of Noun Phrases in Written Discourse. In: Bara, B.G., Bucciarelli, M. (eds.) Proceedings of the 27th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Mahwah (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Crossley, S.A., Varner, L.K., Roscoe, R.D., McNamara, D.S. (2013). Using Automated Indices of Cohesion to Evaluate an Intelligent Tutoring System and an Automated Writing Evaluation System. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds) Artificial Intelligence in Education. AIED 2013. Lecture Notes in Computer Science(), vol 7926. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39112-5_28
Download citation
DOI: https://doi.org/10.1007/978-3-642-39112-5_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39111-8
Online ISBN: 978-3-642-39112-5
eBook Packages: Computer ScienceComputer Science (R0)