1 Introduction

Recently, Japan has been experiencing a declining birthrate and an increasingly aging population; as a result, the number of dementia patients is increasing. Current medical science has no way to treat dementia completely after onset. Therefore, it is necessary to detect mild cognitive impairment (MCI) in the early stage just before dementia develops. If treatment or prevention can be done early in the MCI stage, there is a possibility that dementia will not develop. However, MCI is diagnosed subjectively by a doctor. Recent quantitative research indicates that MCI patients who exhibit subtle deficits in daily living behavior (in this study, micro-errors (MEs)) have declining cognitive function associated with cognitive impairment. Sara et al. demonstrated that there was a high possibility that participants experiencing a large number of MEs had lower cognitive function [1].

In recent years, virtual reality (VR) technology has been actively utilized in rehabilitation and therapy, and VR applications corresponding to human physical and cognitive functions have been developed [2]. In this study, we use an application known as Virtual Kitchen (VK) [3] to analyze how ME occurs. VK is used as an application for participants to perform daily living tasks, such as making breakfast, in virtual space [4]. VK is equipped with a leap motion sensor, and we are able to record finger movements to obtain useful information.

This study utilizes finger movement data and subtask information from VK to analyze how ME occurs. First, we performed data smoothing to reduce sensor noise. Second, we segmented speed data according to subtask completion time. Third, we employed the sliding window algorithm to use Inductive Logic Programing (ILP). Finally, we used ILP to visualize rules and to use qualitative and quantitative data.

2 Finger Movement and VK Data

2.1 Raw Data

The VK system is equipped with a leap motion sensor to collect finger movements and provide two-dimensional coordinate data. This sensor records data at 0.01 per second and contains much noise; thus, data preprocessing is necessary. We performed data smoothing for each dimension using a moving average filter to make data trends easier to understand. Figure 1 plots the results of moving average filter using a span of 50 points. We then calculated speed data from the preprocessed data.

Fig. 1.
figure 1

Result of moving average filter

Full size image

We obtained finger movement data and VK information data that included the state of the finger when touching the screen of the VK application. Thirteen healthy young adults, the pilot participants, prepared breakfast as everyday action tasks in the VK.

2.2 VK Data

Breakfast tasks analyzed in this work consist of preparing toast and making coffee (Table 1). Toast preparation subtasks include putting the toast in the toaster, switching on the toaster, putting the toast on the plate, taking the butter and jelly with knife, and spreading the butter and jelly on the toast. Coffee making subtasks include opening the coffee lid, scooping the coffee, putting the coffee in the mug, opening the sugar lid, scooping sugar with a spoon, stirring the coffee, putting in milk, and stirring the coffee. Figure 2 depicts the configuration of the VK system and the leap motion devices.

Table 1. Subtasks of toast and coffee preparation
Full size table
Fig. 2.
figure 2

System configuration of VK application [3]

Full size image

2.3 Segmentation by Subtask

In this study, we segmented completion time by subtask and used segmented data as sample data for the data-mining method. The time frame in Fig. 3 represents finger speed until completion of the breakfast task. For example, the leftmost frame separates the speed of the fingers until finishing subtask toast 1. The yellow segments indicate that ME was observed during the subtask.

Fig. 3.
figure 3

Segmentation

Full size image

We found 45 positive examples that represent ME events in the subtasks and 168 negative examples of events other than ME.

3 Method

Our methodology proposes a combination of inductive logic programming (ILP) and the sliding window algorithm. Because ILP can extract expressive rules using background knowledge but ILP is susceptible to noise and memory hog, it is difficult to use sensor data directly for ILP. Sliding window is used to reduce the amount of data retaining shape of original time series data. The details are discussed in the next section. To use ILP, we must define some background knowledge that is obtained from finger speed and subtask information: speed, difference in speed, time intervals, the current subtask, the previous subtask, and the next subtask. In the final process, we use ILP to extract rules covering positive data. We used the parallel ILP system known as GKS [5, 6].

3.1 Sliding Window Algorithm

The sliding window algorithm is a piecewise linear approximation algorithm that is often used for machine learning and data mining for preprocessing sensor data [7]. In this study, it is used as it has the ability to reduce the amount of data while holding the shape of original time series data. We used the sliding window algorithm with interpolation against each segmented data [7]. Figure 4 plots the result of the sliding window algorithm with interpolation to reduce noise.

Fig. 4.
figure 4

Example of sliding window result with interpolation

Full size image

In this study, sliding windows are used on the original finger speed data in an effort to produce the smallest number of time points that will be used in ILP background knowledge.

3.2 ILP

ILP is a machine learning method that uses logic as the input. This method can extract rules explaining positive samples but not negative samples, and is a method to express time series data using flexible background knowledge. We defined two categories of predicate: movement and subtask. Table 2 lists predicates of ILP background knowledge. The movement category represents the quantitative data related to finger speed. The speed predicate represents the speed at a certain point in time. The diff_span predicate has two constants that exhibit difference between speed at a certain time point and speed at the adjacent time point, and time intervals between them. The subtask category includes predicates that explain qualitative data. We defined current subtask, previous subtask, and next subtask. By combining finger movement and subtask information, we can determine trends in type of subtask and type of movement when ME occurs.

Table 2. Predicates and mode declarations in background knowledge
Full size table

ILP cannot deal with a continuous value; thus, we must transform data into discrete values. Table 3 lists descriptions of data transformation. The + symbol denotes an input variable, the – symbol indicates an output variable, and the # symbol denotes a constant. We transformed speed into high, middle, and low. We transformed the difference value into three stages so as to be equal frequency, depending on whether it was positive or negative. We transformed time intervals into long, middle and short.

Table 3. Descriptions of transformed values
Full size table

For example, in diff_span, we express change of speed as

The upper expression indicates that time series point 1 is low speed. The middle expression indicates that from time series point 1 to time series point 2 there is a high speed increase at short time intervals. The second expression indicates that from time series point 2 to time series point 3 there is a speed decrease at medium time intervals. We can express the time series change as described above.

4 Results

As a result, we obtained rules using the ILP learning result. “{T, F}” means the number of positive examples and the number of negative examples explained by the rule. The obtained results from ILP are as follows.

  • Rule 1:

ME occurs when there is a high speed increase in a short time interval from the middle point, a moderate decrease occurs at a short time interval, and speed rises greatly in short time intervals.

  • Rule 2:

ME occurs when there is a moderate speed decrease at a middle time interval from the middle point, speed increases moderately in a short time interval, and then it increases greatly in a middle time interval.

  • Rule 3:

ME occurs when there is a moderate speed decrease at a short time interval from the previous time series point; then it increase moderately at short time interval after performing subtask toast 2.

  • Rule 4:

ME occurs when there is a high speed increase at a short time interval from the middle point; then speed decreases moderately at a middle time interval before performing subtask toast 3.

  • Rule 5:

ME occurs when there is a high speed increase at a short time interval, and then speed decreases moderately from the low point in a short time interval in the middle of subtask coffee 2.

Figure 5 presents these results in a time frame.

Fig. 5.
figure 5

Result of rules

Full size image

5 Discussion

From the preliminary experiments, we obtained some rules related to speed change and time interval. Rule 1 and Rule 2 indicate that ME tends to occur when speed rapidly increases and decreases in a short time interval. Also, regarding subtasks, Rule 3 and Rule 4 indicate that ME occurred after toast 2 (when the participant changed from toast task to coffee task) and before toast 3 (when the participant changed from coffee task to toast task). These results indicate that ME occurs during changing of subtasks, when there is much cognitive load. Rule 5 indicates that ME occurred in the middle of performing coffee 3.

Some MEs occur when there is no change in finger speed. Therefore, MEs can be roughly divided into two groups: those that involve speed change and those that involve no speed change. In the group that involves speed change, speed tends to increase and decrease in a short time interval, and ME occurs at the time of task switching.

However, in this study, the number of positive examples and the number negative example are unbalanced, because the amount of positive data was small and only a few samples were covered by the rule. In the future, we will work with more samples to obtain better results.

6 Conclusion and Future Work

This study used ILP to extract rules for subtle deficits occurring during performance of cognitive tasks in a VR environment. We arranged raw data and transformed data into discrete values for use in ILP. We defined background knowledge such as speed, difference, time interval, and subtask; then we used ILP to learn rules that define the occurence of ME. We obtained rules regarding finger movement and subtask in relation to ME occurence. ME tends to occur when movement changes in a short time interval (less than 0.5 s) and when the participant switches from one task to another. We obtained some preliminary results that explain how ME is generally related to subtask and finger speed. However, we had only a small amount of positive data in this pilot experiment. In the future, we will use more positive samples and conduct more experiments to obtain better and more accurate results.