Abstract
It is common practice to compare gain scores in order to determine the effectiveness of adding features to a training system. Here we argue that relying on one measure of overall system effectiveness may result in overlooking valuable lessons available from a comparison of different versions of a system. To illustrate our point, we present the results of comparing a Natural Language Processing (NLP) based adaptive feedback system to a system that does not utilize NLP capabilities. We show that, while there were no learning gain differences between the two systems, the correlates to gain were different. In the non-NLP system, only student performance during the training was correlated to learning gain. In the adaptive system, more variables correlated with learning, including measures of system capability and student satisfaction. This level of analysis suggests that the two systems are not equivalent and points us towards modifications that may improve effectiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Van Lehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46(4), 197–221 (2011)
Dzikovska, M.O., Bental, D., Moore, J.D., Steinhauser, N.B., Campbell, G.E., Farrow, E., Callaway, C.B.: Intelligent tutoring with natural language support in the Beetle II system. In: Wolpers, M., Kirschner, P.A., Scheffel, M., Lindstaedt, S., Dimitrova, V. (eds.) EC-TEL 2010. LNCS, vol. 6383, pp. 620–625. Springer, Heidelberg (2010)
Dzikovska, M.O., Moore, J.D., Steinhauser, N., Campbell, G.: Exploring User Satisfaction in a Tutorial Dialogue System. In: Proceedings of SIGdial 2011 (2011)
Campbell, G.E., Steinhauser, N.B., Dzikovska, M., Moore, J.D., Callaway, C.B., Farrow, E.: The “DeMAND” coding scheme: A “common language” for representing and analyzing student discourse. In: Dimitrova, V., Mizoguchi, R., du Boulay, B., Graesser, A. (eds.) Artificial Intelligence in Education, pp. 665–667. IOS Press, Amsterdam (2009)
Steinhauser, N.B., Campbell, G.E., Taylor, L.S., Caine, S., Scott, C., Dzikovska, M.O., Moore, J.D.: Talk like an Electrician: Student dialogue mimicking behavior in an Intelligent Tutoring System. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 361–368. Springer, Heidelberg (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Steinhauser, N.B., Campbell, G.E., Dehne, S., Dzikovska, M.O., Moore, J.D. (2013). System Comparisons: Is There Life after Null?. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds) Artificial Intelligence in Education. AIED 2013. Lecture Notes in Computer Science(), vol 7926. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39112-5_98
Download citation
DOI: https://doi.org/10.1007/978-3-642-39112-5_98
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39111-8
Online ISBN: 978-3-642-39112-5
eBook Packages: Computer ScienceComputer Science (R0)