Keywords

1 Introduction

Adaptive instructional systems (AISs) are used in formal and informal educational contexts to support student learning. These systems make use of a student/learner model to keep a representation of the student that is deployed by the system to create adaptive interactions. This adaptivity can take the form of different type and amount of feedback, content sequencing, and access to educational materials offered to the student based on the status of the student model. Information to initialize and keep the student model up-to-date can come from different sources (e.g., student process and response data).

No matter the type of AIS, one must keep in mind the need for providing students and other educational stakeholders with the information they need to support their decisions. However, many AISs are built as “black boxes,” using data driven approaches that provide limited information for students, teachers, parents, or other stakeholders [1,2,3].

This paper elaborates on the need to support human inspection of the student model (e.g., student performance levels) and some of the inner workings of AIS (e.g., evidence used to infer such performance levels). That is, to guarantee transparency AISs should provide educational stakeholders not only with information about student progress or mastery levels, but also with information about how the decisions made by the system were based on relevant evidence and how these decisions supported student learning.

2 Open Student Models

Research in open student models (OSM) has shown that allowing students to inspect or interact with a representation of the student model can reduce the complexity of the diagnostic and selection processes and at the same time support student learning and self-reflection [4,5,6].

Creating open student models that support learning can be challenging, partially due to the various types of student data that can be represented in a model, and the variety of ways people might interact with that model. The student model can include various types of information about the student such as: cognitive abilities, metacognitive skills, affective states, personality traits, learner styles, social skills, and perceptual skills. These sources of student model information can vary in terms of their quality and granularity.

Several approaches to interacting with open student models have been explored. Some of these approaches include the use of external representations such as text, progress bars, concept maps, mastery grids, hierarchical graphs, and directed acyclic graphs, and interaction approaches like following interaction protocols, guided exploration using an artificial agent, collaborating with a peer, and negotiating the model with a teacher or a virtual tutor [7,8,9,10,11].

Multiple audiences can benefit from the student modeling information maintained by AISs. Audiences such as students, parents, teachers, and researchers are potential users of this information. While the type of representation and interaction mode may differ according to the needs, and other characteristics of each audience, the AIS should have capabilities to support access and facilitate interpretation of the information maintained by the system. That is, AISs should be able to support the creation of dashboards or reporting systems that can provide each audience with the information they need to support their decisions.

3 An Interpretation Layer for AISs

AISs integrate and make sense of these diverse pieces of student information to make adaptive decisions that positively influence student learning. However, AISs do not always implement mechanisms to support the creation of reporting systems or dashboards that provide information to students and other stakeholders (e.g., teachers and other third parties interested in knowing more about the inner workings of the AIS).

Inspired by assessment design methodologies such as evidence-centered design (ECD; [12]), which makes use of argument-based structures to articulate the chain of reasoning connecting task-level data to the evidence required to support assessment claims about the student, Zapata-Rivera et al. [13] proposed the implementation of an evidence layer for intelligent tutoring systems. This evidence layer can facilitate the creation of adaptive systems by maintaining sound argument-based structures that represent how student data are used to support assessment claims about the student. This layer also aims at formalizing the concept of evidence, facilitating reuse of evidence, supporting the integration of claims and evidence in reporting information produced by the tutor, and aiding with the automatic evaluation of evidence.

Tools and services can be implemented on top of this evidence-layer to create an interpretation layer that can support the creation of reporting systems, dashboards and open student models.

Zapata-Rivera et al. [13] show how the internal evidence-layer in conversation-based assessments [14] can be used to identify and reuse pieces of evidence (e.g., quality of description of an earth science process, accuracy of a selected sequence of events associated with the process, accuracy of identification and quality of description of an event based on data collected, quality of selection between two data collection notes, and accuracy of prediction based on data collected) across two different systems designed to assess science inquiry skills. Also, the authors elaborate on applications of this approach to improve the use of evidence in military prototypes that were implemented using the Generalized Intelligent Framework for Tutoring (GIFT; [15, 16]).

Technologies such as the experience application programming interface (xAPI; [17] can support the implementation of common data structures that various AISs can use to share user information. By adopting an assessment vocabulary similar to the one offered by ECD, it is possible to implement an evidence layer for AISs [18].

4 Evidence-Based Interaction with Open Student Models

Given the richness of process and response data available in AISs, different interactions, approaches, and external representations could be employed to provide users with the information they require. The existence of an interpretation layer for AISs, such the one described above, would enable the implementation of open student modeling approaches that not only provide users with information about their current knowledge levels and other characteristics, but also provide them with information about the evidence used to support these values.

An example of such an approach to open student modeling is called evidence-based interaction with OSM (EI-OSM; [6]). In this approach, a user interface based on Toulmin’s argument structure [19] is used to provide the student with information about current assessment claims maintained by the system and corresponding supporting evidence. Assessment claims are assertions regarding students’ knowledge, skills and other abilities (e.g., a student’s knowledge level on a concept).

Figure 1 shows an assessment claim and supporting evidence regarding Clarissa’s knowledge of “Calculate slope from points.” The assessment claim with the highest strength value appears at the top of the screen. The strength of an argument represents a measure of the credibility, relevance, and quality of the supporting evidence for each side of the argument. Supporting evidence for the system claim appears on the left, while the student’s alternative explanation and available evidence appears on the right. This type of interaction can be part of a formative dialogue between teachers and students (e.g., what type of evidence should be provided to strengthen the student’s argument. Also, the AIS can provide suggestions about learning materials and tasks that can be administered to update the current state of the student model.

Fig. 1.
figure 1

The student suggests an alternative explanation to the claim “Clarissa’s knowledge level of Calculate slope from points is Low” and provides supporting evidence.

Full size image

The effectiveness of interactive OSM and external representations to clearly communicate the information and support decisions should be evaluated with the intended audience. EI-OSM was evaluated with eight teachers who interacted with various use cases (i.e., understanding and using a proficiency map, exploring assessment claims and supporting evidence; and assigning credibility and relevance values and providing adaptive feedback) and provided feedback during small group interviews (1–3 teachers and at least 2 interviewers per session) [6]. Each interview lasted for about 2 h. Some of the results of this study included: (1) teachers appreciated the use of evidence-based argument structures due to their potential for improving communication with students and parents. However, interviewees mentioned that some teachers may not have enough time and resources to implement this approach; teachers provided suggestions to facilitate the implementation in real settings (e.g., generating email alerts for teachers to inform them about particular situations that may require their attention, and involving tutors and teacher assistants in the process to divide the work); (2) teachers would like the system to use algorithms to handle common cases but want to be in control in case they need to override the system’s actions; teachers appreciated the option to use their own instructional materials and tasks but suggested the use of predefined instructional packages of materials and task banks that can be integrated automatically into the system; finally, (3) some teachers mentioned possible issues with students trying to game the system by adding unsupported claims but recognized that the system could support learning and help improve self-assessment skills by engaging students in a goal oriented process that involves gathering evidence (e.g., by learning about particular topics and solving relevant tasks) that can be used to strengthen their assessment claims.

5 Designing and Evaluating Reporting Systems

Research on score reporting may provide some insights into the type of work that can be done in order to better support human inspection of AISs:

  • Work on frameworks for designing and evaluating score reports [20, 21]. These are iterative frameworks involving activities such as gathering assessment information needs, reconciling these needs with assessment information available to be shared with the audience, designing prototypes based on design principles from areas such as information visualization, human computer interaction and cognitive science, and evaluating them with the intended audience [22]. These frameworks also take into account the client expectations, whether the report is designed in a research vs. an operational context, and propose the use of various evaluation approaches including cognitive laboratories, focus groups and large-scale studies evaluating the effectiveness of alternate score reports.

  • Following standards on the quality of assessment information (e.g., psychometric properties of scores and subscores) to decide whether or not the information provided can support appropriate uses of the results [23, 24].

  • Designing and evaluating score reports targeted for particular audiences (e.g., teachers, parents, students and policymakers) in summative and formative contexts. This work includes the use of simple reports using traditional graphical representations and interactive report systems or dashboards. Evaluation of these reports usually includes examining comprehension and preference aspects [25, 26].

  • Developing supporting materials such as tutorials, interpretive and ancillary materials that offer guidance on the appropriate use of assessment results [27].

This work as well as work on designing and evaluating OSM [10] can inform the development of AISs that consider the information needs of different types of users, which may positively impact the acceptance and extensive adoption of AISs in a variety of educational contexts. Table 1 shows sample assessment information needs for various types of users.

Table 1. Sample assessment information needs for various types of users.
Full size table

6 Discussion

In this section we discuss several recommendations and challenges to improving support for human inspection of AISs.

  • An interpretation layer. As presented above, by relying on an evidence layer, it is possible to implement an interpretation layer that supports the creation of OSM, reporting systems and dashboards targeted for particular audiences. As more AISs are built using machine learning approaches that may be difficult to inspect [1,2,3], it is important to think about ways of interpreting the results or recommendations made by these models and make them available for the creation of systems that support the needs of users. For example, some students may want to know why a particular problem or piece of feedback was presented by the AIS [6, 28, 29]. Also, teachers, administrators and researchers can be interested in knowing how students interact with the AIS and how their data are used by the system to adapt its interaction. Approaches for interpreting “black box” models are being explored [30,31,32].

  • Iterative design and evaluation frameworks. AISs should follow well-established design and evaluation frameworks and standards that aim at producing systems that offer high quality information at the right level to support the decision-making needs of target users including students, teachers, parents, administrators and researchers. Designing and evaluating interactive systems that clearly communicate AIS information to these users should be part of the development cycle of any AIS. Lessons learned from evaluating the use of external representations with particular audiences should be made available so that designers can benefit from these types of results. It is not enough to simply design a dashboard that shows all the data available in the system. Dashboard components should be evaluated to guarantee that users appropriately understand and use the information presented [33].

7 Future Work

Future work in this area includes the development of platforms that implement interpretation layers; the use of iterative, audience-centered frameworks for the design and evaluation of OSM, reporting systems, and dashboards; additional work on communicating information from machine learning approaches that are difficult to interpret; and the creation of standards that serve as guidance for the type of information that should be available to support user decisions in AIS.