How to Explain It to a Model Manager?

Abstract

In the context of explainable AI (XAI), little research has been done to show how user role specific explanations look like. This research aims to find out the explanation needs for a user role called “model manager”, a user monitoring multiple AI-based systems for quality assurance in manufacturing. The question this research attempts to answer is what are the explainability needs of the model manager. By using a design analysis technique (task questions), a concept (UI mockup) was created in a controlled way. Additionally, a causal chain model was created and used as an assumed representation of the mental model for explanations. Furthermore, several options of confidence levels were explored. In a qualitative user study (cognitive walkthrough) with ten participants, it was investigated which explanations are needed to support understandability, trustworthiness, and actionability. The research concludes four findings: F1) A mental model for explanations is an effective way to identify uncertainty addressing explanation content that addresses target user group specific needs. F2) “AI domain” and “application domain” explanations are identified as new explanation categories. F3) “show your work” and “singular” explanations are identified as new explanation categories. F4) “actionability” is identified as a new explanation quality.

A Appendix

1.1 A.1 Screener Questions

In your current job:

• Do you work with AI solutions? (general fit)

• Do you aim to optimize the up-time of an AI-solution? (reflection of user goal UG 1)

• Do you aim to ensure a defined quality goal of an AI solution? (reflection of user goal UG 2)

• Do you identify quality deviations of an AI solution? (reflection of user task UT 1)

• Do you troubleshoot quality deviations of an AI solution? (reflection of user task UT 2)

• Do you check that the troubleshooting of an AI solution was successful? (reflection of user task UT 3)

• Do you respond to customer requests? (reflection of user task UT 4)

1.2 A.2 Concept

View 1 (Fig. 2 in Appendix) shows a list of automatically reported incidents. Each incident is described with several attributes. One attribute is an incident number to identify an incident. The second attribute is a customer name. We assume that a model manager is responsible for several customers and incidents have been reported from one customer. The third attribute is the host which is a hardware device on which the incident occurred. Each incident has a category. In our examples, we distinguish here between model, hardware, and software incidents. An incident has a date and a time of occurrence. Furthermore, an incident has a criticality level that indicates the severity of an incident. An incident has a status and an owner.

Fig. 2.

View 1: Overview about reported incidents

The following views show details of the incident with the incident number IN-0001 (the first one of the incident list). On the left-hand side, key incident attributes are shown, as they have been used in view 1. The incident category has a more refined description. On our case, “hidden feature draft” was added as a detail. The incident has four detailed views, represented by views 2 through view 5. View 2 shows the symptom (see also “symptom” in Fig. 1) and view 3 shows the suggested causes (see also “root cause 1” and “root cause 2” in Fig. 1). View 4 shows suggested actions (see also “suggested action 1” and “suggested action 2” in Fig. 1) and view 5 shows the efficacy of the selected and initiated action.

We now look at each view more in detail. View 2 (see Fig. 3 in Appendix) shows the symptom of the incident. The symptom is from the AI domain. In our example, the f1-score was selected as the metric for the symptoms. Other metrics can be used, too. The detailed view show that the f1-score dropped from 61% to 37% at a certain date and time.

Fig. 3.

View 2: Symptom

View 3 (see Fig. 4 in Appendix) shows suggested causes. On the left-hand side, one or several suggested causes are displayed, and they are sorted by a automatically determined confidence level. The suggested cause with the highest confidence level is listed first. The detailed view for a suggested cause shows root cause 1 (application domain) and root cause 2 (AI domain). For the root cause 1, a time series chart is displayed to visualize when the event occurred. In our case, the event was the change of the soldering paste. A mapping of the root cause 1 time series chart (change of soldering paste) and the symptom time series chart (drop of f1-score) is explicitly mapped (see dotted vertical lines, marked as 3.4). The suggested cause (root cause 1, application domain) and the root cause 2 (AI domain) are described.

View 4 (see Fig. 5 in Appendix) shows suggested actions. The suggested actions refer to one of the suggested causes. One element in view 4 shows how the criticality of the incident is determined (see element 4.3). The area with suggested actions (element 4.4) is separated into two parts. The first part shows an immediate action to reduce additional damages from the symptom and the suggested causes. The second group are suggested actions that have the intend to resolve the selected suggested cause. Several suggested actions can be listed and ranked by their confidence level. For each suggested action, the action, the estimated time, the return of investment is shown, including people that need to be involved when a suggested action is selected to be initiated.

Fig. 4.

View 3: Symptom and suggested causes

Fig. 5.

View 4: Suggested actions

Fig. 6.

View 5: Action efficacy

After an initiated action was fully executed, View 5 (see Fig. 6 in Appendix) shows whether the action was effective. Effective means that it has resolved the root cause and that the symptom has disappeared. View 5 is separated into two parts. In the top part (elements 5.3 and 5.4), the efficacy of the initiated action is summarized. The event overview (element 5.5) shows all events along several time series charts with synchronization lines across the charts (element 5.6). The efficacy chart top bar is a summary of the sequence of events that provides a binary visual clue whether the action was successful (green check mark, as shown here) or not (red cross, not shown here). It also shows the key start event (root cause 1) and finish event (last action), the change of the metric values (here f1-score) and a graphical overview about the f1-score across the timeline. The event overview shows for the application domain and the AI domain all events across the timelines and synchronization lines for application events (change of soldering paste) and AI solution events (f1-score dropped, “old” model retired, “new” model launched in shadowing mode).

Fig. 7.

View 6: Confidence level, options 1, 2, 3a, 3b, 3c

Fig. 8.

View 6: Confidence level, options 4a, 4b, 4c

1.3 A.3 Confidence Level Options

Option 1 does not show a confidence level to the user at all. The system automatically selects the item with the highest confidence level, without further explanations. Option 2 shows several suggested items without a confidence level for the user. The item with the highest confidence level is selected automatically. An additional explanation for each item is not given. Option 3 uses a qualitative confidence level, with three variations. Option 3a shows a qualitative confidence level (e.g., high, medium, low) and a short explanation from the application domain is given. Option 3b shows a qualitative confidence level with a short explanation from the AI domain. Option 3c shows a qualitative confidence level with a short explanation from the application domain as well as from the AI domain (see Fig. 7 in Appendix).

Another option shows a confidence level with a quantitative confidence level, with three variations. Option 4a shows a quantitative confidence level as a percentage value and a checklist from the application domain. The relative number of met checkpoints determines the percentage value. Option 4b shows a quantitative confidence level with a checklist from the AI domain. Option 4c shows a quantitative confidence level with a checklist from the application domain as well as from the AI domain (see Fig. 8 in Appendix).

1.4 A.4 Questions

Topic 1: Understandability

• Question 1.1: Which element(s) of any views help you best to understand what happened and why? (Answer options: no element, one or multiple elements)

• Question 1.2: Which view helps you best to understand what happened and why? (Answer option: one selected view (out of five presented views))

• Question 1.3: Why have you selected this view?

Topic 2: Trustworthiness

• Question 2.1: Which element(s) of any views help you best to trust the outcome of the application? (Answer options: no element, one or multiple elements)

• Question 2.2: Which view helps you best to trust the outcome of the application? (Answer option: one selected view (out of five presented views))

• Question 2.3: Why have you selected this view?

Topic 3: Initiate Action

• Question 3.1: Which element(s) of any view help you best to initiate a responsive action? (Answer options: no element, one or multiple elements)

• Question 3.2: Which view helps you best to initiate a responsive action? (Answer option: one selected view (out of five presented views))

• Question 3.3: Why have you selected this view?

Topic 4: Check Efficacy of Initiated Action

• Question 3.1: Which element(s) of any view help you best to check the effectiveness of the initiated action? (Answer options: no element, one or multiple elements)

• Question 3.2: Which view helps you best to check the effectiveness of the initiated action? (Answer option: one selected view (out of five presented views))

• Question 3.3: Why have you selected this view?

Topic 5: Mapping of Application Domain Explanations to AI Domain Explanations

• Question 5.1: Rate the following statement: The mapping of the AI solution information to the application domain information is important for me to understand what happened and why. (Answer options: 5-point Likert scale)

• Question 5.2: Rate the following statement: The mapping of the AI solution information to the application domain information is important for me to understand that the initiated action was effective or not. (Answer options: 5-point Likert scale)

Topic 6: Validate the Causal Chain Model as the Assumed Mental Model

• Question 6.1: The causal chain model reflects the key elements of an incident very well. (Answer options: 5-point Likert scale)

• Question 6.2: In addition to the responsive action: Which other content do you need to see to understand what happened and why per incident? (Answer options: root cause 1, causal chain, root cause 2, responsive action, none, something else; multiple selections are possible)

• Question 6.3: In addition to the responsive action: Which other content do you need to see to trust the outcome of the application? (Answer options: root cause 1, causal chain, root cause 2, responsive action, none, something else; multiple selections are possible)

Topic 7: Confidence Level

• Question 7.1: Which option to present one or more system selected options do you prefer? (Answer option: one confidence level option)

• Question 7.2: Why have you selected this option?

• Question 7.3: Rate the statement: The selected option for showing and explaining the confidence level is important for me to understand what happened and why. (Answer options: 5-point Likert scale)

• Question 7.4: Rate the statement: The selected option for showing and explaining the confidence level is important for me to trust the outcome of the application. (Answer options: 5-point Likert scale)