poster

Semantic Matching Evaluation of User Responses to Electronics Questions in AutoTutor

Authors:

Colin M. Carmon,

Andrew J. Hampton,

Arthur C. GraesserAuthors Info & Claims

L@S '19: Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale

Article No.: 36, Pages 1 - 4

https://doi.org/10.1145/3330430.3333649

Published: 24 June 2019 Publication History

Abstract

Relatedness between user input and an ideal response is a salient feature required for proper functioning of an Intelligent Tutoring System (ITS) using natural language processing. Improper assessment of text input causes maladaptation in ITSs. Meta-assessment of user responses in ITSs can improve instruction efficacy and user satisfaction. Therefore, this paper evaluates the quality of semantic matching between user input and the expected response in AutoTutor, an ITS which holds a conversation with the user in natural language. AutoTutor's dialogue is driven by the AutoTutor Conversation Engine (ACE), which uses a combination of Latent Semantic Analysis (LSA) and Regular Expressions (RegEx) to assess user input. We assessed ACE via responses from 219 Amazon Mechanical Turk users, who answered 118 electronics questions broken into 5202 response pairings (n = 5202). These analyses explore the relationship between RegEx and LSA, agreement between the two judges, and agreement between human judges and ACE. Additionally, we calculated precision and recall. As expected, regular expressions and LSA had a moderate, positive relationship, and the agreement between ACE and human was fair, but slightly lower than agreement between human.

References

[1]

Zhiqiang Cai, Arthur Graesser, Carol Forsyth, Candice Burkett, Keith Millis, Patricia Wallace, Diane Halpern, and Heather Butler. 2011. Trialog in ARIES: User Input Assessment in an Intelligent Tutoring System.

[2]

Colin Carmon, Andrew Hampton, Brent Morgan, Zhiqiang Cai, and Arthur Graesser. 2018. Semantic matching evaluation in ElectronixTutor. In Proceedings of educational data mining 2018 11, 580--583. Buffalo, NY.

[3]

Fabiano Dorca. 2015. Implementation and use of simulated students for test and validation of new adaptive educational systems: A practical insight. International Journal of Artificial Intelligence in Education 25, 319--345.

[4]

Myroslava Dzikovska, Natalie Steinhauser, Elaine Farrow, Johanna Moore, and Gwendolyn Campbell. 2014. BEETLE II: deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. International Journal of Artificial Intelligence in Education 24, 284--332.

[5]

Dipesh Gautam, Zachari Swiecki, David Shaffer, Arthur Graesser, and Vasile Rus. 2017. Modeling classifiers for virtual internships without participant data. In Proceedings of the 10th International Conference on Educational Data Mining 10, 278--283. Wuhan, China: EDM Society.

[6]

Arthur Graesser, Zhiqiang Cai, Brent Morgan, and Lijia Wang. 2017. Assessment with computer agents that engage in conversational dialogues and trialogues with learners. Computers in Human Behavior.

Digital Library

[7]

Arthur Graesser, Sidney D'Mello, Xiangen Hu, Zhiqiang Cai, Andrew Olney, and Brent Morgan. 2012. AutoTutor. Applied Natural Language Processing: Identification, Investigation and Resolution 169--187.

[8]

Arthur Graesser, Xiangen Hu, Benjamin Nye, Kurt VanLehn, Rohit Kumar, Cristina Heffernan, Neil Heffernan, Beverly Woolf, Andrew Olney, Vasile Rus, Frank Andrasik, Philip Pavlik, Zhiqiang Cai, Jon Wetzel, Brent Morgan, Andrew Hampton, Anne Lippert, Lijia Wang, Qinyu Chen, Joseph Vinson, Craig Kelly, Caddarius McGlown, Charvi Majmudar, Bashir Morshed, and Whitney Baer. 2017. ElectronixTutor: an intelligent tutoring system with multiple learning resources for electronics. International Journal of STEM Education: Innovations and Research.

[9]

Diane Halpern, Keith Millis, Arthur Graesser, Heather Butler, Carol Forsyth, and Zhiqiang Cai. 2012. Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity 7, 93--100.

[10]

Stephen Ivens and Bertram Koslin. 1991. Demands for Reading Literacy Require New Accountability Methods. Touchstone Applied Science Associates.

[11]

Thomas Landauer, Danielle McNamara, Simon Dennis, and Walter Kintsch. 2007. Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.

[12]

Alan Lesgold, Susarine Lajoie, Marilyn Bunzo, and Gary Eggan. 1992. SHERLOCK: a coached practice environment for an electronics trouble-shooting job. In Computer assisted instruction and intelligent tutoring systems: Shared goals and complementary approaches 201--238. Hillsdale, NJ: Erlbaum.

[13]

Elizabeth Liddy. 2001. Natural Language Processing. In Encyclopedia of Library and Information Science, (2nd Ed.) NY. Marcel Decker, Inc.

[14]

Benjamin Nye, Arthur Graesser, and Xiangen Hu. 2014. AutoTutor and family: a review of 17 years of natural language tutoring. Int J Artif Intell Educ, 24, 427--469.

[15]

Andrew Olney, Sidney D'Mello, Natalie Person, Whitney Cade, Patrick Hays, Clair Williams, Blair Lehman, and Arthur Graesser. 2012. Guru: a computer tutor that models expert human tutors. In Proceedings of Intelligent Tutoring Systems (ITS) 2012, 256--261. Berlin: Springer.

Digital Library

[16]

Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring systems. Educational Psychologist 46, 197--221.

[17]

Kurt VanLehn, Greg Chung, Sachin Grover, Ayesha Madni, and Jon Wetzel. 2016. Learning science by constructing models: can Dragoon increase learning without increasing the time required? International Journal of Artificial Intelligence in Education 26, 4, 1 36.

Cited By

Carmon CHu XGraesser A(2023)Assessment in Conversational Intelligent Tutoring Systems: Are Contextual Embeddings Really Better?Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky10.1007/978-3-031-36336-8_19(121-129)Online publication date: 30-Jun-2023
https://doi.org/10.1007/978-3-031-36336-8_19

Semantic Matching Evaluation of User Responses to Electronics Questions in AutoTutor
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Electronixtutor integrates multiple learning resources to teach electronics on the web
L@S '18: Proceedings of the Fifth Annual ACM Conference on Learning at Scale

ElectronixTutor is a new Intelligent Tutoring System for electronics that integrates multiple intelligent learning resources, including AutoTutor, Dragoon, LearnForm, ASSISTments, and BEETLE-II, as well as Point & Query hotspots on diagrams and numerous ...
AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back
Special issue on highlights of the decade in interactive intelligent systems

We present AutoTutor and Affective AutoTutor as examples of innovative 21^st century interactive intelligent systems that promote learning and engagement. AutoTutor is an intelligent tutoring system that helps students compose explanations of difficult ...
Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor's Mixed-Initiative Dialogue

This paper investigates how frequent conversation patterns from a mixed-initiative dialogue with an intelligent tutoring system, AutoTutor, can significantly predict users' affective states (e.g. confusion, eureka, frustration). This study adopted an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '19: Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale

June 2019

386 pages

ISBN:9781450368049

DOI:10.1145/3330430

Copyright © 2019 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2019

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

L@S '19

L@S '19: Sixth (2019) ACM Conference on Learning @ Scale

June 24 - 25, 2019

IL, Chicago, USA

Acceptance Rates

L@S '19 Paper Acceptance Rate 24 of 70 submissions, 34%;

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
181
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Carmon CHu XGraesser A(2023)Assessment in Conversational Intelligent Tutoring Systems: Are Contextual Embeddings Really Better?Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky10.1007/978-3-031-36336-8_19(121-129)Online publication date: 30-Jun-2023
https://doi.org/10.1007/978-3-031-36336-8_19

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten