Keywords

1 Introduction

Approximately one in five adults aged 16 or older in thirty-three of the OECD (Organization for Economic Cooperation and Development) countries have literacy skills at a low level of proficiency [19]. A challenge for literacy centers is that low literacy adults have heterogeneous characteristics such as age, race/ethnicity, country of origin, highest educational level attained, literacy skills, interests, and goals [2]. The diversity of this population makes it difficult for teachers to differentiate instruction to optimal levels to meet the needs of a group or classroom of students. A computer program, on the other hand, can use learner responses to adapt instruction for each student and get closer to this optimal level [4, 13, 25]. Besides diversity in skill level, another challenge for literacy programs is high absenteeism and attrition rates due to unstable work hours, transportation difficulties, and childcare issues [9, 10, 16, 22]. A web based literacy program can also help with this issue. Being able to access a computer program on the Internet is an excellent way for absent adult learners to continue to improve their reading abilities. It provides students more choice with when and where they choose to work on their literacy skills. These reasons and others are why many believe computer programs can offer a solution for the challenges faced by adult literacy centers.

Web based reading programs may indeed be a feasible solution, especially considering that computers equipped with the internet are becoming more common in literacy centers and in adult learners’ homes. In United States, 1200 federally funded adult literacy programs were surveyed between 2001 and 2002, and results indicated that 80% of the programs used computers in some type of capacity with adult learners [24]. According to the 2003 National Assessment of Adult Literacy [14], 67% of adults who read at grade levels 3 to 7.9 had a computer in their home with Internet access. Computers with Internet access are also available for adult learners in public libraries, children’s schools, and adult literacy programs. Newnan (2015) conducted a survey on more than 1000 programs and found more than 80% of survey respondents had computers in their classrooms with consistent access to the Internet (although significant variability was noted) [17]. In addition, it was reported that an increasing number of adult literacy programs are infusing technology into their classrooms and curriculum [21]. As part of the effort to increase computer-based instruction, we developed an intelligent tutoring system (ITS) called AutoTutor to help adult learners improve reading comprehension skills. In this paper, we introduce three distinctive features of AutoTutor and report the effectiveness of AutoTutor based on the early results of a reading comprehension intervention using AutoTutor.

2 Unique Features of AutoTutor

AutoTutor is a conversation-based intelligent tutoring system (ITS) shown to promote learning on a wide range of topics [5, 18]. Research reports average learning gains of 0.8 σ for AutoTutor compared to various traditional teaching controls [18]. AutoTutor teaches literacy skills by holding conversations in natural language. Conversations between a human student and a pedagogical agent are based on the expectation-misconception tailored (EMT) approach [7]. In this approach, a human student provides answers to questions asked by the agent. The AutoTutor system uses this response to assess a learner’s understanding of the content by comparing it to expected answers or misconceptions in real time. Using this EMT approach, AutoTutor is constantly assessing the students by providing feedback, hints, pumps, prompts to guide the learning through the content.

Traditional AutoTutor systems implement conversations called dialogues that model the interactions that occur between a single human tutor and human student. More recent versions of AutoTutor may employ trialogues which are tutorial conversations between three actors - a teacher agent, a human learner, and a peer agent [6]. Trialogues are implemented in AutoTutor for CSAL, an ITS that was developed by researchers at the Center for the Study of Adult Literacy. The system was designed to help adults with low literacy acquire strategies for comprehending text at multiple levels of language and discourse. The AutoTutor curriculum has 35 lessons that focus on specific comprehension components [5]. Each AutoTutor lesson takes 10 to 50 min to complete. The lessons typically start with a 2–3 min video that reviews a comprehension strategy. After the review, the computer agents scaffold students through the learning by conversation.

2.1 Easy to Use Interface

Before designing the system, a study was conducted to assess the digital literacy of 105 adult learners who read between 3rd and 8th grade. The assessment consisted of a behavioral test and a self-reported questionnaire. For the behavioral test, the participants were presented with tasks that the user needed to complete in a simulated computer interface. For example, if the task is to drag a file to the Recycle Bin, the user must actually click and drag on a file in the simulated interface and then release the file over the Recycle Bin. The behavioral tasks covered various types of computer skills from four categories: Basic Computer Skills, World Wide Web, Windows, and Email. The questionnaire asked the participants about frequency of computer use and computer habits. Results showed the percentage of tasks the adults could perform correctly ranged from 20% to 96%. The tasks in which the adults were least proficient (<30% correct) involved right clicking, typing, and knowledge of different web browsers. The majority of learners were not able to complete simple tasks such as opening a Word document in a taskbar, typing in a web address and clicking NEXT, or choosing a secure password and typing it in a “re-enter password” box [20].

Because of the limited digital literacy skills of adult learners, AutoTutor for CSAL requires minimum use of keyboard input. The system relies heavily on point & click (or touch) interactions, multiple-choice questions, and drag & drop functions. The system does include some writing components that require semantic evaluation of open-ended student contributions, which is the signature feature of AutoTutor systems [5, 18]. In addition, AutoTutor has many pictures, visual displays, easy to read diagrams, and multimedia that help keep the engagement of the adult learner. The system has the capability of reading texts aloud when the learner asks for such assistance by clicking on a screen option. This feature is helpful because many adult learners have limited decoding and word identification skills) [22]. A typical AutoTutor lesson starts with a short tutorial called a “nutshell” which gives the learner a brief visual and audio overview of the lesson topic. This tutorial can be viewed before a user begins a lesson, but may be accessed at any point during the lesson if the learner clicks the “watch video” button on the bottom of the screen. After the tutorial, the computer agents briefly introduce reading strategies related to the topic of the lesson by having a conversation between themselves. Next, readers are provided with the learning material including articles, images, words, etc. Figure 1 is the screen the read can see at the end of an article. At this point, readers are given different choices as to how to proceed. They can click on the “Repeat” button and the agents will repeat the introduction about how to move to the next section and what next section is about. They can press “Play Video” button and the tutorial video will be played again. In addition, there is a green voice button at the bottom right. One of the agents will read the article to the reader when it is clicked. There is a scroll bar to the right side of the learning material, and readers can always drag it up and down to see the parts they want to go over. If the readers are ready to proceed to the next session, they can click the continue bar.

Fig. 1.
figure 1

AutoTutor Interface with multiple interaction elements.

Full size image

2.2 Three-Way Interaction Between Computer Agents and Human Learner

Besides an interface built for people with low digital literacy, another distinctive feature of AutoTutor for CSAL is the trialogue [6]. A trialogue refers to a three-way conversation that takes place between two computer agents (a tutor and a peer) and a human learner. This type of conversational design provides instruction on reading comprehension strategies, help the learner apply these strategies as well as assess the learner’s performance on applying these strategies. Trialogues and conversational dialogues with agents are effective in improving learning on a variety of subject matters and strategies [5, 18].

For most AutoTutor lessons, 10–35 multiple choice questions associated with the learning materials are provided to learners to apply the reading strategies. The questions usually have three alternative answers. A question is asked by the teacher agent (Cristina), and both the learner and the peer agent (Jordan) are required to answer the question. Typically, a learner gets two opportunities to answer a question. In the first trial, if both the learner and peer agent answer the questions correctly, the teacher agent will tell them that their answer is correct and explain why that answer is correct. If the leaner answers the question correctly but the peer agent (i.e. Jordan) answers the questions incorrectly, the teacher agent tells Jordan that he is wrong and the learner is correct. The teacher agent also explains why Jordan’s answer is wrong. If the learner fails to answer the question correctly in the first trial, the teacher agent will let the learner answer it again. After the learner clicks the answer, the teacher agent will give the feedback about whether his/her answer is correct or incorrect as well as the reasons behind the correct and incorrect answers. The example shown by Fig. 2 illustrates how the two agents guide the learner through the question.

Fig. 2.
figure 2

A trialogue between computer agents and human learner with the focus on comparison and contrast.

Full size image
  1. (1)

    Cristina (Tutor Agent): Sam and Joran, now we have read the passage, let’s compare the careers of Michael Jordan and Coby Bryant.

  2. (2)

    Jordan (Peer Agent): I like Kobe more, even though my name is Jordan too.

  3. (3)

    Cristina: Oh! You’re too much! Let’s see if we can find some similarities and differences between them. Sam, click the sentence that shows how Kobe Bryant and Michael Jordan are similar.

  4. (4)

    Sam (Human Student): [Click answer (2)]

  5. (5)

    Cristina: Jordan, is that right?

  6. (6)

    Jordan: Kim, I would have to disagree. It is incorrect.

  7. (7)

    Cristina: Jordan, you got it right. Excellent!

  8. (8)

    Jordan: I think the word “different” in the second sentence is a signal word. This sentence shows a difference, not a similarity.

  9. (9)

    Cristina: Sam, please have another try. Sam, which sentence shows the similarity between Coby and Jordan?

  10. (10)

    Sam (Human Student): [Click answer (1)]

  11. (11)

    Cristina: Sam, you are correct. Nice job! Signal words can help us to see the comparison. In the first sentence, use of the word Two signals they have that information in common. In sentence 2, the word Different indicates a contrast.

  12. (12)

    Jordan: The first sentence says Kobe and Jordan are two of the greatest shooting guards. I think this is a similarity between them.

Apart from guiding and scaffolding learners through the questions, there are motivational features implemented in AutoTutor through competitions between the learner and the peer agent. The learner and peer agent take turns answering questions and score points in the competition which is guided by the tutor agent. Sometimes the learner wins and sometimes the peer agent wins, but ultimately the adult learner will win the competition, no matter how poorly the adult learner performs. This type of competition is expected to boost the adult learners’ self-confidence as well as keep them engaged.

2.3 Adaptivity Through Branching

According to Goldilocks Principle, learning materials and tasks should not be too hard or too easy, but at the right level of difficulty for the students’ level of skill or prior knowledge [23]. It was found individuals are most motivated by challenging tasks that provide an intermediate probability of success, instead of the ones that offer certain success or failure [11]. For adult learners, sometimes it is important to challenge them by assigning more difficult texts, and sometimes they need a boost in self-confidence by receiving easy texts they can readily comprehend.

To select appropriate texts that vary at difficult level, a computational tool called Coh-Metrix [8, 12] was used to scale texts on difficulty with respect to multiple levels of language and discourse by analyzing characteristics of words, syntax, discourse cohesion and text category. There are five major dimensions of Coh-Metrix which are listed and defined below.

Narrativity.

Narrative text tells a story, with characters, events, places, and things that are familiar to readers/listeners. Narrativity is closely affiliated with everyday oral conversation. This robust component is greatly affiliated with word familiarity, world knowledge, and oral language. Narrativity is contrasted by informational (or non-narrative) texts on less familiar topics.

Syntactic Simplicity.

Syntactic simplicity reflects sentences with fewer words and in simpler, familiar syntactic structures which are comparatively easier to process and understand. Difficult sentences have more words positioned in complex, unfamiliar structures, which increase the difficulty of comprehension.

Word Concreteness.

Texts are easier to process if they contain content words that are concrete and meaningful and evoke mental images and are more meaningful. Abstract words increase the difficulty to construct visual representations in the mind and make texts more challenging to understand.

Referential Cohesion.

Texts with high referential cohesion contain words and ideas that overlap across sentences and the entire text, forming threads that connect the explicit textbase. Texts with low referential cohesion are more difficult to process and understand because there is information gap between sentences.

Deep Cohesion.

Texts with causal and intentional connectives help the reader form a more coherent, explicit, and deeper understanding of the text at the level of the causal situation model. Ideas that are related semantically also contribute to deep cohesion. When texts contain many relationships but lack those connectives, inference is required to process the relationships between ideas in the texts.

Graesser et al. [12] identified a composite measure called formality based on the five dimensions, which can be used as a single approximate index of text difficulty. Coh-Metrix formality decreases with narrativity, syntactic simplicity and word concreteness, but increases with referential cohesion and deep cohesion. It was reported that the correlation between formality and Flesch-Kincaid grade levels scores was 0.72 and the correlation between formality and Lexile scores was 0.66 [12].

In a reading comprehension intervention study conducted in Atlanta and Toronto, over 250 adult learners were recruited. After taking pretests that measured their prior reading skills, the adults participated in approximately 100 h of hybrid classes which consisted of teacher-led sessions and AutoTutor sessions. After the intervention, participants completed posttests that assessed their reading skills [3]. While studying with AutoTutor, the logs of students’ online learning activities were recorded by the system. The log file included learner information, class information, lesson and question information, response time and learning outcome. There were 11 text-based lessons used in the study. We analyzed the formality of texts in the 11 lessons and divided them into high formality versus low formality group by median split. The average time on the items associated with high formality texts was found to be significantly longer than that of low formality texts (F (1,11757) = 335.7, p < .001), as is shown in Fig. 3. Meanwhile, the average proportion correct on the items associated with high formality texts was found to be significantly lower than that of the texts of lower formality (F (1,11757) = 49.46.7, p < .001), which is indicated in Fig. 4.

Fig. 3.
figure 3

Time on items associated with texts of high and low formality.

Full size image
Fig. 4.
figure 4

Accuracy of items associated with texts of high and low formality.

Full size image

Most of the text-based lessons of AutoTutor for CSAL have three texts at different difficulty level measured by text formality. The lessons start out providing a learner texts being medium in difficulty at the initial phase. After the learner finishes the tasks associated with the medium level texts, the system will evaluate the learner’s reading skill by comparing the accuracy of the tasks at initial phase to the threshold (i.e. 0.67 in most lessons, whereas the chance performance was 0.33). If the accuracy reaches or exceeds the threshold, the learner will be branched into a condition where more difficulty learning materials will be assigned subsequently. If the accuracy fails to meet the threshold, the learner will be branched into a condition where easier learning materials will be assigned subsequently. Adult learners will get into the phase where the learning material and associated tasks are in balance with their reading skill through branching.

3 Effectiveness of AutoTutor

Fang et al. [3] identified four types of adult readers (i.e. proficient readers, struggling readers, conscientious readers and disengaged readers) based on their learning behaviors interacting with AutoTutor in the reading intervention study. We examined the pretest and posttest scores of the four types of readers in order to see how and to what extent they benefited from AutoTutor. As is shown in Table 1, the learning effect sizes range between 0.23 and 0.64. A meta-analysis reported the effect size of ITS in the domain of language and literacy was 0.34 [15]. The average effect size of the reading intervention using AutoTutor was 0.47, which is higher than that of ITS. In addition, it was found that conscientious readers, proficient readers and disengaged readers benefited more from AutoTutor than struggling readers. This is a common finding in ITS research. Low domain knowledge or low skill students appear to have a harder time learning within ITS than intermediate to high domain knowledge or skill students [15].

Table 1. Means, standard deviations and effect sizes of four types of adult learners.
Full size table

4 Discussion

We developed AutoTutor to teach reading comprehension strategies to low literacy adults using conversational agents. The learners are guided through the learning process by having conversations with a teacher agent and a peer agent. Considering the distinctive characteristics of the user group, we added unique elements in the design of the system. The early results indicated that AutoTutor is a promising tool to help low literacy adults improve reading skills. However, individual differences should be considered when we implement AutoTutor. Adult learners were found to behave differently while interacting with the system [3]. Their benefit from the system also varied. For example, there were conscientious readers who spent extra amount of time on task and achieved high accuracy. As a result, their learning gain was the highest compared to other types of readers. Meanwhile, there were struggling readers who also spent a large amount of time on task but they were wheel-spinning and ended with minimal learning gain. It is possibly because the learning materials were too difficult for them. The disengaged readers were not willing to spend much time on the tasks and their performance was not stable across tasks. These learners might have clicked through some questions when they were not engaged by the tasks. Moving forward it would be useful to improve the adaptivity of AutoTutor by assessing learners in a more nuanced way. Specifically, we can tailor the learning materials and feedback to meet the various needs of the individuals with low literacy skills.