Keywords

1 Introduction

Virtual reality (VR) as defined by [1] is “a scientific and technological domain, exploiting computer science and behavioral devices, allowing a person to interact multi-modally with a virtual world and its 3D entities that interact in real time by means of sensorimotor channels.” As virtual reality has gained traction in the social sciences, scholars have begun to explore its viability for creating novel stimuli, treatments, and learning environments for use outside of the laboratory [2]. VRs have also been explored as a tool for cognitive behavioral therapy [2]. The use of VR-based technology in therapy is increasing in clinical settings because in some cases the technology is well-matched to the needs of clinical applications. In addition, the information recorded and captured by VR-based technology is real-time and can be advantageous for further analysis of patients’ characteristics. Based on experiments and theoretical considerations, when VR-based technology is used in the learning process, useful data can be recorded automatically by software and can contribute to an analysis of each patient’s state [3].

The purpose of the present study was to utilize data collected from VR-based software used for learning in Mild Cognitive Impairment (MCI) cases to examine whether such data can be successfully used in the task of generating rules of action transition in some types of errors made in the everyday actions of older adults. In this work, fourteen older adults completed a breakfast preparing task as an everyday action task in a designated virtual reality called the Virtual Kitchen (VK), equipped with a leap motion controller to record their finger motion. We took video recordings of the task and analyzed the motion-capture data, using these as inputs for inductive learning and using Inductive Logic Programming to produce rules for the errors.

The remainder of this paper is organized as follows. Section 2 reviews earlier work related to errors and slips in daily activities and summarizes the problem description of the study. Section 3 defines the experiments and data set used for analysis. Section 4 presents the process of ILP learning, including data preparation procedures. The resulting learned rules are provided in Sect. 5. Finally, the conclusion and further discussion are provided in the last section.

2 Problem Description

When doing daily activities such as making the bed, taking a shower, or preparing a meal, the process of action does not always go smoothly. For the most part, whether such goal-oriented tasks are completed with ease and according to a plan depends on the robustness and flexibility of human information processing. However, even in the most familiar of tasks, errors or slips of action do occur at a nontrivial rate [4].

Theories concerning the origins of action slips have been proposed by some preceding works [5,6,7] using the term “micro slips.” Micro slips refer to the microscopic regulation of behavior. They are commonly observed in everyday sequential activities [8]. The first experimental research on micro slips was done by Reed and Schoenherr [9] and resulted in four classifications of micro slips: hesitation, trajectory change, touch, and hand shape change. These findings have been used in succeeding works on micro slips, especially on how a trajectory change in the hand has a significant relationship with the execution of micro slips. Some other works [10, 11] attempted to improve the classification of micro slips using three transition types. Nevertheless, the classification depended heavily on the direction of movement and did not give due weight to action transitions during sequential activities [8].

The study in [8] defined some new definitions for micro slips, using “motion” as a new coding scheme. Motion is a component of a sequential behavior and represents the manner in which a hand movement acts on an object. According to the scheme, micro slips are cases in which the motion does not proceed smoothly until the task is finished; it often changes into another motion along the way. In the study results, five basic transition patterns were defined, depending on the specific action transition of the hand towards the object.

Such slips and errors have attracted the interest of psychologists and neuropsychologists [7, 12], and more recently they have become a focus as a predictor in analyzing the potential for Mild Cognitive Impairment (MCI) in older adults. For example, [13] presents a novel measure of slip and errors (called micro errors) in doing daily activities in order to improve the coding and monitoring of errors as an indicator of MCI. They defined five categories of micro errors: Reach – Touch errors, Reach – No touch errors, Reach with Object errors, Extra Action errors, and Sequence errors. They found that the micro errors could be reliable and sensitive, indicating their potential as a valid index of MCI. However, to date, there is no study examining how the new definition of micro errors relates to the direction of the hands nor to the hand action transition, which the present authors believe would be useful for better understanding in characterizing MCI patients.

This study has the goal of generating rules on the nature of micro errors by collecting finger movement and analyzing the basic transition patterns of study participants doing daily activity tasks in a virtual reality-based environment. Another purpose is to explore the use of analyzing finger speed when micro errors occur. In this study, the micro errors that will be used in the analysis are limited to two types: Reach – Touch (RT) errors and Reach – No Touch (RNT) errors.

3 Data Set

For this study, we asked fourteen (n = 14) older adults to complete the task of preparing a breakfast (breakfast task) in the VK environment as seen in Fig. 1. In performing the breakfast task, participants were expected to prepare a piece of toast and a cup of coffee. Toast preparation subtasks include putting the toast in the toaster, switching on the toaster, putting the toast on the plate, taking the butter and jelly with knife, and spreading the butter and jelly on the toast. Coffee-making subtasks include opening the coffee lid, scooping the coffee, putting the coffee in the mug, opening the sugar lid, scooping sugar with a spoon, stirring the coffee, putting in milk, and stirring the coffee.

Fig. 1.
figure 1

(a) Configuration of the virtual kitchen environment [14] (b) Screen shot of the virtual kitchen breakfast task [14]

Full size image

From 14 older adults, based on a number of clinical tests and participants’ historical data, four of them meet the criteria for Mild Cognitive Impairment, and three of them did not have MCI when the experiments were performed but did have MCI in the past.

Fourteen video clips were obtained for analysis from the breakfast preparing tasks. The basic motion units in the breakfast preparing task were determined based on preliminary observation of these video clips and adapting the work in [8]. There are five essential motions (select, take, open, scoop, pour) and no motion (pause). The finger movements all through the task were coded by watching the video in terms of these motion units. Uncompleted motions were coded using “-ing” in order to differentiate them from completed motions. Micro errors were coded over the whole task and classified into two types (reach - touch (RT), reach - no touch (RNT)), as defined in Table 1. A total of 116 micro errors was observed throughout the video clips, including 93 RNT errors and 23 RT errors (Table 2).

Table 1. Definition of micro errors [13]
Full size table
Table 2. Definition of motions coded from video
Full size table

4 Methodology

An overview of the methodology used in this study is presented in Fig. 2. We obtained several types of data as an input based on the interaction between the user and the Virtual Kitchen device, we then employed some preprocessing and learning to generate rules for the micro errors.

Fig. 2.
figure 2

Methodology overview

Full size image

4.1 Motion Data Preprocessing

The VK system is equipped with a leap-motion sensor to collect finger movements and provide two-dimensional coordinate data. Data preprocessing of the motion data to obtain speed data is performed by data smoothing along each dimension using a moving-average filter to make data trends readable. After we calculated the speed from the motion data, we integrated the quantitative and qualitative motion data, executing the following steps.

Step 1. :

Collect a set of speed data and average each attribute value of the motion data

Step 2. :

Detect motion that reveals a micro error event and obtain a set of time codes

Step 3. :

Integrate the data from Steps 1 and 2 with the finger-motion state coding

Step 4. :

Add attributes indicating speed differences at the times when a finger motion is started and completed

Step 5. :

Integrate the data obtained in Step 4 into qualitative data (upLow, upMiddle, upHigh, downLow, downMiddle, downHigh)

The error detection in Step 2 and the finger-motion state coding in Step 3 were performed manually at the current stage. Detailed definitions of the subtasks are given in Table 4 in the Appendix. Table 3 gives an example of data integration after performing the five steps mentioned previously.

Table 3. Example on data integrating
Full size table

4.2 ILP Learning

Like other machine learning methods, ILP algorithms take examples of certain concepts, in our case related to two classes of micro errors, in order to construct a hypothesis that generalizes (the terms ‘explains’ or ‘covers’ are also used) the examples. Domain-specific knowledge, also called background knowledge, may be added in order to simplify the learning process; this kind of knowledge is given and does not have to be learned [15]. In this study, we used an ILP system called GKS [16, 17] to employ ILP and generate rules.

Coding micro-error as examples for ILP

Suppose, for example, that we want to learn the definitions of two types of micro-errors such as Reach – Touch and Reach – No Touch. As our aim is to recognize motion transitions in micro errors, the first task is to collect examples of these micro errors as well as motion examples associated with normal motion when no micro errors occur. Second, since ILP is a symbolic machine-learning method, we must generate a formal description of these micro errors.

To use symbolic data in ILP, the first task is to define a formal language in order to abstract the numerical data. The choice of language used in ILP is important because it can influence the learning results [15]. We chose to represent the temporal samples as a set of motion descriptions. A series of motions A is denoted with the following arguments:

  • Subtask: a constant symbol set by looking at which subtask the motion belongs to.

  • Motion: the state of hand motion; select, take, open, scoop, pour, pause.

  • Speed difference: the qualifying value of finger speed while doing the motion; low, middle, high (both for positive and negative values of speed change).

  • Time: the qualifying value of time length of a motion; short, middle, long

  • Before: the name of the preceding subtask

  • After: the name of the next subtask

The fourth and sixth arguments (before and after) specify even chaining in the finger motion. They are considered to provide important information as they reveal the symbolic temporal relationships between the elements of motion.

5 Learned Rules

We performed two experiments in using ILP for learning. First, we used only Reach – Touch errors as positive examples, not considering any Reach – No Touch errors, and therefore all examples other than RT errors are negative examples. In the second experiment, we used only Reach – No Touch errors as positive examples with all examples other than RNT errors considered as negative examples. This process was used in order to get a better result for the prediction rules. We present the rules that are the most readable and are related to the real conditions of the experiments below. “[T, F]” denotes the number of positive examples (T) and the number of negative examples (F) the rule covers. “rt” represents the presented rules for the Reach – Touch errors, and “rnt” represents the presented rules for the Reach – No Touch errors.

5.1 Reach – Touch Error Rules

Several rules related to the Reach – Touch errors are listed below.

Rule 1

A Reach – Touch error happens if there is a pause motion (no motion of finger) after performing subtask “Toast 3,” which is turning the toaster on, and there is a decrease in speed for a medium time range of the pausing motion.

Rule 2

A Reach – Touch error happens if there is a “take” motion before performing subtask “Cream 1”, which is selecting cream, and there is an increase in speed for a long time range of the selecting motion. For example, after finishing whichever task was before the cream task, the participants tend to take an unwanted object that is not related to the previous or subsequent task.

5.2 Reach – no Touch Error Rules

Several rules related to Reach – No Touch errors are listed below.

Rule 3

A Reach – No Touch error happens if there is a moderate speed increase over a middle time interval when performing action “take” after performing “Coffee 8”, which is stirring the coffee. For example, after stirring the coffee in the mug, a participant may tend to select an unwanted object that is not related to the succeeding task.

Rule 4

A Reach – No Touch error happens if there is a small speed decrease over a long time interval when performing no motion or pause before performing “Jelly 1,” which is selecting jelly.

Rule 5

A Reach – Touch error happens if there is a selecting motion after performing subtask “Toast 4”, which is taking toast to the plate, and there is a decrease in speed over a medium time range of the selecting motion. For example, after putting the toast on the plate, a participant may select an unwanted object that is not related to the succeeding task.

6 Conclusions and Discussion

We applied ILP to generate rules on how two types of micro errors happen in the task of learning in the virtual-reality environment called Virtual Kitchen. We described how to acquire motion data and transform the data into qualitative data that is useful for ILP learning. The focus of this study was to see how action transitions of the finger in the experiments were related to the pattern of micro errors.

For Reach – Touch errors, based on the learned results, the first rule suggests that there is a tendency for a micro error to happen after a medium-length pause. This rule naturally describes how participant motion is related to cognitive activity. When they pause, there seems to be a cognitive process going on while deciding what to do in the next step. Looking at the subtasks, the participants seem to make errors after the type of task in which they have to change the subtask (for example, from toast to coffee), which is supported by the fact that slips or errors tend to happen in between the decision points.

For Reach – No Touch errors, two rules represent how such errors tend to happen under the motion of selecting. This is also supported by the tendency of making errors at the decision point, because in subtask segmentation selecting is always put at the start of each subtask, meaning that the participants make errors just before they start to make a correct step at the beginning of small subtasks.

This study, however, has some limitations, including the fact that the task of action coding was still done manually. Also, the limited number of positive examples makes it seem that the rules cover only a few positive examples. Future work includes executing the action coding automatically, using available data from the virtual-kitchen device, and focusing on how to generate rules that can differentiate between patterns of micro errors involving healthy older adults and MCI patients