Keywords

1 Introduction

The Generalized Intelligent Framework for Tutoring (GIFT) is an open-source, modular architecture developed to reduce the cost and skill required for authoring adaptive training and educational systems, to automate instructional delivery and management, and to develop and standardize tools for the evaluation of adaptive training and educational technologies (Sottilare et al. 2012a, b). By separating the components of ITSs, GIFT seeks to reduce development costs by facilitating component reuse.

Meta-analyses and reviews support the claim that intelligent tutoring systems (ITS’s) improve learning over typical classroom teaching, reading texts, and/or other traditional learning methods. (Dynarsky et al. 2007; Dodds and Fletcher 2004; Fletcher 2003; Graesser et al. 2012; Steenbergen-Hu and Cooper 2013, 2014; VanLehn 2011). In fact, ITS’s have been shown to improve learning to levels comparable to Human tutors (VanLehn et al. 2007; VanLehn 2011; Olney et al. 2012).

While improved training effectiveness is certainly a benefit of ITS technology, another important benefit is improved training efficiency over one-size-fits-all training. The goal of an ITS is to identify the gaps in knowledge specific to each learner so that training can focus on filling just those gaps. One of the problems of one-size-fits-all training is that to insure all trainees can comprehend the instruction, it must be developed for trainees with the least experience, knowledge, and aptitude. Though less costly to develop, the material is presented a pace that is slow and that includes content not needed for more experienced, higher aptitude trainees. An ITS would be expected to reduce the time needed to deliver training to such trainees.

The reduction in time to train (i.e., improved acquisition rate) is an important metric because reductions in training time represent cost savings. This is especially true for military trainees who are paid a salary. Reductions in the time needed to train those trainees saves salary costs for both trainees and instructors. For large-volume courses, those savings can be substantial.

All of this highlights the need for a means to model and predict training efficiency gains by ITSs generally and GIFT specifically. Having the ability to model time saved by the use of adaptive, intelligent training, as compared to existing or non-adaptive training would have benefits throughout the lifecycle of a course. During the design of new training, the training developer could more easily make decisions about the relative costs and benefits of adding adaptive features. For example, adding extensive remedial training for easy-to-understand concepts may benefit such a small percent of the population of learners, that the net reduction in training time would be too small to make those features worth the cost of development.

During training delivery, actual trainee data could be used to verify and/or improve the model. For example, suppose the model assumed that learners with an aptitude above criteria A would have a 95% probability of understanding concept B without needing any remediation. Learner data could then be used to validate or adjust that probability. This improved model could then be used to better determine the true time-savings of the course when delivered by GIFT.

During training evaluation and refinement, the disparity between predicted and observed training outcomes could be used to refine the training. For example, if a segment of training proves to be more difficult than anticipated for a group of learners, it is possible that the training segment should be refined or redeveloped.

An example of such a model was developed by McDonnell Douglas (1977). This model incorporated predictor variables in four broad categories: course content (e.g., difficulty, length of content), instructional design (e.g., instructional strategies/techniques), test characteristics (e.g., difficulty, number of items), and trainee characteristics (e.g., aptitude, motivation). The model predicted about 39% of the variability in trainee’s first-attempt lesson time for self-paced computer-based instruction.

To understand how GIFT might begin to model and predict training time for learners, it is necessary to understand how training is adapted by this system. GIFT is a framework that modularizes the common components of intelligent tutoring systems. These components include a learner module, an instructional or tutor module, a domain module, and a user interface. One of the main motivations for creating this framework was to lower the cost and labor needed to create intelligent tutoring systems by facilitating re-use of components and by simplifying the authoring process (Sottilare et al. 2012a).

GIFT adapts training using the learning effects model depicted in Fig. 1 below. At the first point of this model, learner data informs learner state in the learner module. The learner module receives assessments from both sensors and the domain module. The learner state is used to determine the appropriate instructional strategy by the tutor module. The instructional strategy is then interpreted by the domain module and used to determine the domain specific learning activities needed to instruct the learner in that domain. The responses of the learner to that activity then update the learner module which starts the cycle over again.

Fig. 1.
figure 1

The learning effects model

Full size image

As can be seen, developing a predictive model in GIFT is not a straightforward process given the ways that training is adapted to each individual. We should note that our goal is not to predict the single path that a trainee would be expected to take through a specific course, but rather the probability associated with all possible paths through the training for a given learner. From that we can determine the range and distribution of times that would be expected for that learner to complete the training. Taking this one step further, we could apply this to a population of learners and predict the range and distribution of the time for that population to complete that training.

The development and integration of a probabilistic model for predicting time to train into the GIFT architecture is currently in the first phase of a three phase plan. In this paper, we describe work being done in the first phase. In this phase we are developing the structure of the Bayesian probabilistic model, identifying factors that are expected to impact training time, and mapping those to a specific course delivered by GIFT. In the second phase, we will integrate this model into the GIFT framework and develop the user interface to allow for authoring of new predictive models for other GIFT courses. In the third phase of the work, we will empirically validate the predictive model in GIFT and make adjustments to try to improve it.

2 Methods

This section describes our method for modeling adaptive training content and predicting distributions of completion times for both individuals and groups using the GIFT excavator trainer. This course is available with public version of GIFT. The training content includes text, images, video demonstrations, and practice opportunities in a virtual simulator.

First we describe an example adaptive training course in GIFT, the excavator training. Second, we describe the approach to modeling this content of adaptive training using the Methodology for Annotated Skill Trees (MAST). And finally, we describe our approach to developing probabilistic models of trainees executing the task, including static variables (e.g., prior knowledge and expertise in the domain) and dynamic variables (e.g., fatigue and boredom) which may be observed or latent.

2.1 An Adaptive Training Course in GIFT: Excavator Training

The excavator training course consists of MS PowerPoint slides that have text and text questions as well as a 3D simulation environment for practice. The excavator training starts with a welcoming message and a set of survey questions that extract the learner characteristics of motivation, grit, and self-regulatory ability. The GIFT tutor, then, presents the concepts of rules to control the excavator (i.e., Excavator, Boom, Bucket, Arm, and Swing), and corresponding examples. Figure 2 shows the overall structure of the excavator training contents.

Fig. 2.
figure 2

The overall structure of the excavator training in GIFT.

Full size image

GIFT has been developed to strengthen the capability of adaptive courses. One of recent advances is an implementation of “Adaptive Course Flow” in GIFT (e.g., Sottilare 2014; Goldberg and Hoffman 2015). This was formerly known as the Engine for Management of Adaptive Pedagogy (EMAP) which supports adaptive capabilities for training based on the Component Display Theory (CDT, Merrill 1983). The CDT supports a general framework of skill training that progresses through two types of learning activities, each with two categories: expository (rules and examples) and inquisitory (recall and practice). According to Merrill, learners should progress through these four quadrants in order starting with rules (presentation of general principles), then to examples (presentation of a specific instance), then to recall (declarative knowledge test of the trainee’s comprehension), and finally to practice (opportunity for the trainee to perform the skill). By sorting learning activities into these four quadrants, adaptive training systems like GIFT can apply the CDT to any domain as long as content for that domain is so labeled.

The Adaptive Courseflow (AC) object also considers learner traits and states when determining the most appropriate content to present to the learner in each quadrant. For example, content can be tailored to the motivation, experience, arousal, etc. of the trainee. For example, when initiating the course, students self-identify as either: novice, journeyman, or expert. The learner progresses through the expository quadrants and then is evaluated in the inquisitory quadrants. So, if the learner fails to demonstrate an understanding of the rules or examples in the recall or practice quadrants, the AC object attempts to remediate and reevaluate the trainee before progressing him or her to the next quadrant or lesson. Performance is assessed at either below, at, above expectation.

In the excavator training, rules to control the excavator for each concept (i.e., Boom, Bucket, Arm, Swing) are presented to the learner in the Rule phase, and corresponding examples are presented in the Example phase. In the Recall phase, a batch of assessment questions, shown in Table 1, is presented to the learner in an attempt to identify the learner’s knowledge of each concept. The adaptive behavior of the GIFT tutoring system is dependent on the number of correct answers for each concept as defined by the course author. Within the allowed number of attempts, the learner receives adaptive instructions based on his or her performance. For example, a novice would receive the Rule or Example remediation content, and a journeyman would receive the Example remediation content. The adaptive behavior of the concept remediation occurs up to the total number of three attempts—if the learner fails to reach the anticipated level within the total number of three attempts, the learner is advised to see an instructor. In the practice phase, the learner can practice the acquired knowledge and skills through a practice training application that can be information as text, information from file, or information from web. The excavator training uses the Dynamic Environment Testbed program shown in Fig. 3.

Table 1. Questions used for the knowledge assessment in the excavator training.
Full size table
Fig. 3.
figure 3

The excavator 3D simulation training environment.

Full size image

2.2 Modeling the Content of Adaptive Training

To model the content of adaptive training, we use MAST skill trees. Figure 4 shows a visual representation of a MAST skill tree. The “skeleton” of the skill tree is a procedure model that breaks down entire procedures into constituent steps, tasks, and subtasks. Annotations (shown as colored boxes in the tree and in greater detail on the right) are added to the procedure model; these annotations make MAST unique. For example, consider completing a set of questions in the excavator tutor that features hints and feedback. This step includes tasks for reading the introduction to the problems, each problem, reading hints, and reviewing feedback. Critical for adaptive training, the MAST procedure model represents not only the base procedure of answering each question correctly without hints, but also the optional hints and feedback steps, variations, and multiple potential paths among questions as chosen by GIFT.

Fig. 4.
figure 4

PAST Time uses MAST skill trees to represent adaptive training content (Color figure online)

Full size image

Annotations within the MAST skill tree include the following additional information for each step, task, and subtask (that is, for each skill tree node).

  • Information Elements: Information or knowledge needed by the trainee to perform the actions required by the skill tree node. These requirements are commonly called the “knowledge map” in ITS literature. In the example of completing a set of GIFT questions, this is the knowledge used to answer the question correctly.

  • Instructional Resources: Resources to teach the skills needed to perform the actions required by the node. In the question example, these are pointers to additional training content.

  • Skill Priorities: Ratings of the difficulty and criticality of the skills needed to perform the actions required by the node. These ratings enable training systems to prioritize skills for training and optimize ROI. In the question example, ratings express the criticality of answering the questions correctly to the overall learning goals.

  • Assessments: Methods of assessing the skills required by the node. These methods enable training systems to determine trainee ability. In the question example, assessment methods include secondary measures of trainee cognitive workload, motivation, or affect that may influence completion time.

  • Decision Making Models: Computational models of how the procedure steps, tasks, and subtasks are chosen and ordered. These models enable some of the adaptation logic to be represented in the skill tree. In the question example, these models encode the rules for providing hints, providing feedback, and selecting the next question.

For this effort, we have extended this set of annotations to include a Completion Time Data annotation that describes a distribution of completion time based on past data or an estimate of completion time based on type. This data will be used to train the prediction algorithms. Figure 5 shows a portion of a MAST skill tree for the excavator training GIFT course. This skill tree focuses on the information elements that most heavily influence completion time. On the left, the overall course on Excavator is the root of the tree structure. Its children are the different topics covered by the course, including the Boom Movement topic. This topic features a number of slides with Pictures, Audio, and Text components. Individual trainees may vary in the amount of time they spend examining the Pictures, whether or not they listen completely to the Audio, and the amount of time taken to read the Text. Trainees may also choose to view optional Slides explaining concepts that they may not be familiar with, adding more time. If trainees fail to demonstrate sufficient knowledge in the quiz or fail to complete the simulation tasks appropriately, they are sent back to the beginning of the Boom Movement topic on Slide 1, adding significant time to completion of the course. This model may be expanded to represent a maximum number of failures before the trainee either moves to a different topic or ends the course.

Fig. 5.
figure 5

High-level design of a MAST skill tree of a GIFT module with representations of individual instructional elements, branching content, and variables that influence completion times

Full size image

After reviewing the Slides, the trainees are asked to practice their skills in Simulation. The MAST model of the simulation can be either a complex procedure describing the steps needed to complete the scenario and optional steps that may or may not contribute to the overall goal. The MAST simulation model may also be simple, as shown in Fig. 5 representing just the type of simulation and the number of scenarios. To save modeling time and effort, these MAST models are constructed with only the level of detail needed to sufficiently accurately predict completion time.

2.3 Modeling the Adaptive Training Execution

We use a probabilistic model to represent the different factors and instructional strategies that impact the completion time of a MAST module, as well as probabilistic inference techniques to determine a distribution of course completion time. Not only must our model represent relationships between variables and paths in the MAST skill tree, but it must also recognize and model the impact of time as well; many variables can change as the trainee is completing a training module. Building this model consists of two basic steps: developing a model that estimates completion time for nodes in the MAST skill tree, and temporally linking these models together to enable inference of the entire module completion time.

Figure 6 shows part of an example model for estimating the completion time of a node. This example shows some contributing factors that could be used by PAST Time to estimate the time it takes for a trainee to read the text on the slide. There are also variables that estimate the time to process the pictures and audio on the slide, but that these have been omitted from this example for brevity. The model includes a Reading Time variable, which represents the time it takes for the user to read the text. The value of this variable is a function of the amount of text on the slide, the speed at which the trainee can read the text (Read Speed), and the current alertness of the trainee (Fatigued). These relationships are probabilistic. For example, if a trainee normally reads at 100 words per minute, there are 100 words in the text, and the trainee is tired, the reading time of the trainee could be distribution uniformly from 1 to 2 min. The reading speed of the trainee is also a non-deterministic variable that depends on how much prior knowledge the trainee possesses about the subject, and statistics about how fast the general population of trainees read.

Fig. 6.
figure 6

Example model for estimating the time to read text on a slide node

Full size image

One of the benefits of building a probabilistic model to represent completion time is that not all of the information in the model is needed to estimate the completion time. For example, if we know how much prior knowledge the user has about the subject (for example, from a pre-instruction questionnaire), we can post that knowledge as evidence to the model that would be taken into account when estimating the completion time. If we do not possess that information, we can treat the variable as latent and use a prior distribution to represent the state of the variable. For example, we can estimate that only 20% of trainees taking the course have prior knowledge of the subject. These prior distributions can be estimated from the literature review or expert knowledge, and then learned over time based on the outcomes of actual testing.

Once we determine the probabilistic relationships between the variables in each node, we will develop a dynamic relational model that models the actual temporal process of instruction and the uncertain paths that a trainee may take through the MAST skill tree. Figure 7 shows an example dynamic relational model.

Fig. 7.
figure 7

Dynamic relational probabilistic model that includes variable relationships over time and traversal uncertainty

Full size image

This dynamic relational model is intended to capture two critical elements of predicting completion time: the dynamic process of learning (e.g., becoming fatigued) and the uncertainty inherent in traversing the MAST skill tree. The example shown in the figure shows three states of the learning process. The first state models the completion time to read the text on slide N of a module (this is the same node in Fig. 6, but other variables have been removed for clarity). Once the trainee completes the instruction on the slide, they progress to the simulation node of the module. Here, we model the completion time of the node as the amount of time it takes to complete a scenario in the simulation. The time to play a scenario depends on whether the trainee is fatigued which affects their current cognitive workload. One important aspect of this model is that we create temporal relationships between the variables to model the explicit process of a trainee taking this course. For example, at State 2 in this figure, the probability that a trainee is fatigued increases if the trainee was already fatigued or they spent a significant amount of time completing the previous node.

State 3 in Fig. 7 explicitly shows the relational uncertainty that can impact completion time. In this example, a trainee is required to successfully pass a minimum number of individual training scenarios. We explicitly model the number of passes and failures of a trainee, since each time a trainee engages in a scenario, the completion time is impacted. Therefore, we have a variable that represents whether the trainee succeeded in the scenario, and a variable that represents whether the user is done with the node. If the trainee has not successfully completed the required number of scenarios, then State 3 again models the trainee engaging in a simulation scenario; otherwise, State 3 models the completion time of the next node in the MAST tree (a quiz). This modeling of the relational nature between nodes in the MAST tree is critical for accurate completion time prediction.

Once these probabilistic models are defined, they can be used to compute a distribution over the course completion time. To generate this distribution, a modeler first provides knowledge about a trainee, group of trainees, or a module as evidence to the model. This could be statistical information obtained from the trainees from a pre-course questionnaire, or data obtained from prior training. Then, given the posted evidence, the user can apply standard probabilistic inference techniques (e.g., variable elimination, importance sampling, Metropolis-Hastings, support computation, most probable explanation (MPE), and particle filtering) to generate a distribution over the completion time of the module. These specific methods are included in the Figaro libraries. Statistical moments of this distribution (e.g., mean and variance) can be easily computed and presented to a module designer.

A significant advantage of combining this probabilistic modeling with the MAST skill tree representation is the capability to ascribe time to individual models, and perform “what if” analysis by adding or removing components. For example, a node for a module requiring detailed arithmetic may take little time in and of itself, but it may be fatiguing, causing significant downstream effects in terms of overall training completion time.

3 Results

3.1 Implementing the Adaptive Training Models

The probabilistic model is being implemented using Charles River Analytics’ open source probabilistic programming language, Figaro™ (Pfeffer 2012), to construct and learn probabilistic models of the relationships between these factors. The use of Figaro will greatly simplify the authoring of these models which can be complex and require a high degree of experience by users who may not be experts in probabilistic reasoning.

Creating probabilistic models for specific analytical applications, such as this training completion time prediction problem, presents both representation and reasoning challenges. Figaro enables the easy creation and manipulation of these probabilistic models. Figaro is extremely expressive and can represent a wide variety of models, including:

  • Directed and undirected models with latent relationships, such as trainee motivation

  • Dynamic models with temporal factors, such as trainees completing several nodes of a module

  • Models in which conditions and constraints are expressed by arbitrary functions to represent a wide variety of relationships among the trainee model and adaptive training content

  • Open universe models in which we do not know what or how many objects exist, such as the number of times that a trainee will exercise a particular simulation scenario

  • Models in which the elements themselves are rich data structures, such MAST skill trees

Figaro provides a rich library of constructs to build these models, and provides ways to extend this library to create new model elements. Figaro’s library of reasoning algorithms is also extensible. Current built-in algorithms include exact inference using variable elimination, importance sampling, Metropolis-Hastings with an expressive language to define proposal distributions, support computation, most probable explanation (MPE) using variable elimination, and particle filtering for dynamic models. Figaro also contains built-in learning algorithms based on Expectation-Maximization so that prior distributions can be updated over time. To ease user adoption, PAST Time will leverage Charles River’s previous efforts to provide an easy-to-use graphical interface to build these models that compiles into Figaro programs.

Figure 8 shows an example Figaro program that creates the completion time model for the node slide shown previously in Fig. 6. Note that the probabilities and values in this program are notional.

Fig. 8.
figure 8

Figaro program that models reading time of a slide node

Full size image

First, we define the amount of text in the node as 1000 characters. Then, we define two latent variables, one representing the prior knowledge of the trainee and the other representing typical reading speeds. In this case, we specify that a trainee has prior knowledge with 0.2 probability, and the trainee’s reading speed is normally distributed around 100 characters a second. Next, we define the actual reading speed of this trainee. In this example, if the trainee has prior knowledge of this subject, we increase their reading speed by a value normally distributed around 50 characters a second. We next represent the fatigued state of the trainee (0.4 probability that the trainee is fatigued). Finally, we define the reading time of this node as the amount of text divided by the reading speed of the trainee; if the trainee is fatigued, however, we assume they can only read at 50% capacity. To use this model to estimate the completion time of the module, we use Figaro’s built-in importance sampling algorithm to sample the model and print the distribution over the reading time variable. Observe that invoking an inference algorithm to estimate the completion time is a single line of code, and any other Figaro inference algorithm can be substituted into this program with no other changes.

Figaro probabilistic programming is useful in this context for a number of reasons: We can automatically build a model given a specification of the MAST skill tree, the trainee model, and a set of known relationships. Prediction based on the model is already coded in Figaro’s inference algorithm, so additional effort is not required to use the model. Figaro supports the creation of dynamic Bayesian networks that model the temporal processes of variables, simulating fatigue and practice effects. We can continuously learn using these models; the probabilistic programs are flexible enough to update relationships between variables based on historical or dynamic data. Figaro’s encapsulation mechanism enables easy creation of reusable components. Trainee models and MAST skill trees can be reused for future prediction models. It is embedded in a general purpose language, Scala, which allows the creation of front end graphical interfaces that can edit and invoke the models created in Figaro. Figaro is free and open source, enabling the Sponsor and others to edit, create, and share source code for models.

Figure 9 shows the results of running this Figaro model. The distribution of reading times has three modes. At about 7 s, individuals that have prior knowledge and are not fatigued read the slide quickly. At 10–11 s are individual that have no prior knowledge and are not fatigued. At 20–21 s are individuals without prior knowledge and who are fatigued, reading slowly to absorb more information. An instructor may use a model like this one to examine how individual slide contents may be processed by a class of students, and make small changes to the presentation to increase learning efficiency.

Fig. 9.
figure 9

Probability density of reading times for one slide

Full size image

Figure 10 shows the probability density of reading times over three slides with the student having increased chance of fatigue (40%, 45%, and 50%) on each successive slide. In this simulation, only a small portion of the students are in the fastest group, completing three slides in about 20 s. The bulk of the students range from 25–55 s for these three slides, with three modes in this range covering the combinatorics of prior knowledge and different possible fatigue states on each slide. Also, a significant portion of the students take longer than 55 s, with a possibility of up to 76 s to complete. An instructor can use this model to examine the differential effects of fatigue, prior knowledge, and reading speeds of a heterogeneous group of students, and adjust learning content or course expectations accordingly.

Fig. 10.
figure 10

Probability density of reading times for three slides with increasing chance of fatigue

Full size image

This modeling can reveal underlying properties of adaptive learning content that may be counter-intuitive at first glance. For example, the most likely reading speed of a single slide (according to the first model) is about 10 s. For three slides, one might assume 10 * 3 = 30 s, but the distribution in Fig. 10 shows the mean of the predicted time about 41 s with significant standard deviation. Allotting only 10 s on average per slide in a course would prevent about two-thirds of students from completing all of the course content.

Adaptive training content with significant remedial steps has a much wider variance of completion times. We hypothesize that retraces through previous material (e.g., reviewing the boom operation slides) will be performed much faster than the initial trace. Trainees may also be able to optimize their reading and comprehension strategies if they know how they will be tested and what the consequences for failing are. Therefore, later sections in an adaptive training course (e.g., excavator bucket handling after boom handling) may have significantly different variable interactions than earlier sections, as trainees learn the training structure.

Fatigue, boredom, and other dynamic variables that represent aspects of trainee state may be heavily influenced by elements outside of the tutor, such as the time of day the training is taken, the amount of sleep the trainees received, or the amount of prior instruction (e.g., after a long day of lectures or first learning activity of the day). This may cause a seemingly high degree of variance within the completion times that may be accounted for by measuring these external variables, estimating them from data, or controlling them such that they do not vary among trainees and training instances.

Short training content, such as a course that can be completed in 30 min to an hour, has a significantly different models than extended and repeated training content, such as a course that lasts for multiple hours or content that is experienced over many sessions. Using tools such as the approaches described here can help training staff make intelligent tradeoffs between alternate course structures (e.g., a single 3–4 h session once a week or 3 sessions of approximately an hour per week).

4 Discussion

We believe that including a capability to predict training time for trainees in GIFT has several significant advantages. First, it facilitates return on investment calculations by enabling the author to determine training time reductions resulting from the addition of adaptive features. Second, it provides a means for GIFT to monitor student progress against an expected timeline. Students who take much longer to complete training than expected may not be fully engaged in the training or may be having difficulty with the material. These are conditions that might prompt a response by GIFT. Finally, it can play a role in quality control of GIFT courses. For example, if segments of a course take much longer than expected across multiple trainees, GIFT could flag those sections for review by the course author to insure that the material is presented clearly.

There are several challenges we may face as we move into the second and third phases of this effort. First, the initial MAST skill trees may not contain sufficient variables to predict adaptive training completion times. Our initial literature review and analysis has identified a potential set of most influential variables, but these variables may not be reflective of completion time upon closer inspection. We will mitigate this risk by widening the scope of task models to incorporate more predictive variables if necessary.

Second, while the model predictions may be highly accurate, there is a risk that the system will be too difficult or time consuming to use for some or all of the target populations of instructional designers, course managers, and instructional staff. We mitigate this risk by conducting a requirements analysis early in the effort to closely examine the needs of these user groups and design our system and interfaces to best meet those needs. We will apply human factors and user-centered design and understand the challenges of and methods for developing highly useful and usable decision-aiding tools for practitioners.

Third, while this approach combines state of the art probabilistic approaches and identifies key variables from the literature and past experience, there is a potential that the initial predictions will not sufficiently account for the variability of trainee completion times. We plan to mitigate this risk by incorporating historical data early and adjusting the analysis techniques to capture the maximum amount of variability from data that can be reasonably collected in the field.

When complete, this will be the first system to predict the completion times of GIFT modules from module and trainee data and the first to enable effective assessments of the ROI of key design and implementation decisions for adaptive training systems. It includes an innovative application of procedural skill modeling using demonstrated MAST skill trees to flexibly represent adaptive training content for analysis. It is the first application of Figaro probabilistic programming to predict completion times for adaptive training technologies, including both unobserved latent variables and temporal factors, such as trainee fatigue, boredom, or flow.

The ability to predict the completion time of adaptive training content directly supports the ARL mission and the Army Learning Model (ALM). This approach, if successful, enables instructional designers, course managers, and instructors to make intelligent decisions about adaptive training development, implementation, and use within Army training by providing critical metrics of time to complete training. These metrics significantly impact the ROI of adaptive training and can differentiate between feasible and infeasible training approaches. Optimizing this metric can yield significant cost savings for large Army training courses, where the total cost of trainee time is very high. Tools such as the models described in this paper enable the Army to better understand and streamline adaptive training, as well as invest intelligently in new training technologies by providing accurate estimates of these technologies’ impact on cost and force readiness.