Keywords

1 Introduction

The number of Internet users continues to increase in Japan, especially the elderly. Due to individual differences such as experience and cognitive ability, it is essential that we provide personalized user interfaces, or recommend services that suit the user’s skills. To achieve it, we urgently need efficient method of evaluating the user’s skills. However, unfortunately, the conventional approach of using questionnaires or tests are not user friendly. In order to evaluate the user’s skills without burden and automatically, we focus on the logs of user operations.

Previous studies have examined Web search and estimated user satisfaction [1], success or lostness [2] from the logs of user operations. However, few studies have attempted to estimate skill level from these logs. Our solution is to use machine learning to build estimation models of the skills from operation logs. We focus on the situation in which evermore elderly people will use tablet PCs for online shopping. This study will lead to the personalization of user interface to suit skill level.

2 Definitions of Skills

In this study, we identify the 6 key skills necessary for effective Web search based on previous studies as shown below.

Attitude. Attitude toward technology, in particular, anxiety and self-efficacy affect the breadth of Web use [3]. Having a positive attitude and trust toward the Web are necessary for using Web sites.

Experience. Web experience is correlated with Web search performance [4]. If users use the Web for a long time and frequently, they can search information efficiently by using correct knowledge and an appropriate mental model.

Spatial Ability. Spatial ability is generally defined as the ability to perceive and transform visual patterns [5]. Many previous studies showed that spatial ability is related to Web search performance [4, 6–8].

Processing Speed. Processing speed is defined as the ability to acquire, interpret, and respond to information quickly and accurately. Many previous studies showed processing speed is related to Web search performance [4, 6–8].

Working Memory. Working memory is the active storage and manipulation of information to perform a task. Many previous studies showed working memory is related to Web search performance [4, 6–8].

Motor Ability. Motor ability is the ability to move hands or fingers quickly and accurately. Takahashi et al. reported that motor ability is related to Web search performance [7].

3 Experiment

We conducted an experiment to give participants the labels of skill level and to obtain the operation logs. We recruited participants who were not biased against the use of Information and Communication Technology. 16 elderly people, 10 men and 6 women participated in the experiment. Ages ranged from 65 to 75 years. All participants have a computer, 6 participants have either smartphone or tablet PC. 6 participants have online shopping experience.

Participants operated tablet PC (8.9 inch Android Nexus9, 320 dpi) on their lap. During the experiment, audio and video were recorded for later analysis. The experiment consisted of 3 parts: (1) Operational practice on the tablet PC (2) Skill evaluation task (3) Web search task.

3.1 Skill Evaluation Task

In order to give participants the labels of skill level, evaluation tasks were created and then performed by the participants.

Attitude. Participants completed an Internet attitude questionnaire [9] that was translated into Japanese. Attitude towards Internet was tested using 40 questions with a 4-point Likert-type response scale (for instance, “I have never been frustrated with Internet. Strongly disagree = 1, Disagree = 2, Agree = 3, and Strongly Agree = 4”). A higher score indicated a more positive attitude.

Experience. Participants completed a WWW experience questionnaire [4, 6] translated into Japanese. Experience was measured by 4 questions: the number of hours per week spent using the WWW, the number of years of WWW experience, the number of Web sites developed, and the number of different Web sites visited per week. Principal components analysis was conducted and it produced one significant component that accounted for 80 % of the variance in all the questions. A higher score based on component loadings indicated more experience.

Spatial Ability. Spatial ability was measured using the ETS Cube Comparison test and the ETS Paper Folding test [5]. The Cube Comparison test consists of drawings of pairs of cubes. Participants compare the two cubes and judge whether the two are the same or different. The Paper Folding test consists of drawings of square piece of paper being fold and punched. Participants decided which unfolded paper is correct. The mean of each individual’s normalized score on these tests was used as an index of spatial ability.

Processing Speed. The visual search task was used to measure processing speed as previous study [7]. A double conjunction (Shape: circle or square \(\times \) Color: blue or orange) visual search task was performed in 36 trials on the tablet PC. Participants searched and tapped a target object with a unique feature (for instance, the orange circle is the target in Fig. 1a). If the target was absent, participants tapped an absence button. A target was present in 50 % of trials, and display size was set at a constant 11 elements. Mean reaction time on target-present trials was used as an index of processing speed.

Working Memory. Working memory was estimated with a reading span test following study [7]. Participants were told to read aloud a series of Japanese sentences (2–4 sentences) and remember a target word (bold, underlined) in each sentence at the same time. Task was performed in 10 trials using 30 sentences. The number of words correctly recalled was used as an index of working memory.

Motor Ability. Motor ability was measured on the tablet PC (see Fig. 1b) following study [7]. Participants tapped two targets labeled “1”, “2” in order as fast as possible in 72 trials. Target size was 30, 40, or 50 px, and distance between the two targets was 300, 400, or 500 px. Mean time from target 1 tapped to target 2 tapped was used as an index of motor ability.

3.2 Web Search Task

In order to obtain the operation logs, Web search tasks were created and then performed by the participants. We developed an online shopping Web site for tablet PC use that provides daily necessities (see Fig. 2). The Web site had 4 layer structure and included navigation menu, breadcrumb list and back/forward button. Each item page included a picture, information of price, stock, and rating. In the Web search task, participants were required to compare three items and add the appropriate one to the shopping cart (for instance, “compare prices of noodles, vegetable juice and pen, then add the cheapest one to shopping cart”). There were 2 test trials after practice, and each trial started from the top page. We also developed a custom Web browser that collected the operation logs such as screen touch behavior, accessed Web pages, and menu button interactions. Then, feature values were calculated (see Table 1).

Fig. 1.
figure 1

Skill evaluation task. (Color figure online)

Full size image
Fig. 2.
figure 2

Online shopping web site

Full size image
Table 1. Calculated feature values
Full size table

4 Estimation Model

4.1 Analysis

Skill scores of the participants were calculated from the results of the skill evaluation tasks for the 6 skills. 8 participants whose score was lower than a median were labeled low-skill, while higher than a median were labeled high-skill. Trials in which the participant could not complete a task or ignored instructions in the Web search task were excluded from the analysis. Eventually, 23 trials from 14 participants were used for building the estimation models.

Decision tree-based estimation models of the 6 skills were built by using the labels of skill level and the feature values. Due to the interpretability and simplicity of the CART algorithm, we adopted it as the basis of the decision tree. In order to avoid over-fitting, trees were pruned and the right-sized tree was selected in accordance with the 1 SE rule. Leave-One-Out Cross Validation was used in order to measure the accuracy.

4.2 Results and Discussion

All 6 estimation models and accuracies are shown in Fig. 3. All models are simple and could very accurately estimate skill level, in particular experience and processing speed.

The level of experience was estimated from AccessedPageCount with 86.95 % accuracy. The results showed that the level of experience is high if the count of unique accessed Web pages per second is high. This can be interpreted as saying that users who have more experience move to new Web pages rapidly due to their quick operation based on past experience.

The level of spatial ability was estimated from TotalAccessedPageCount with 69.56 % accuracy. The level of spatial ability is high if the count of total accessed Web pages per second is high. Users who have high spatial ability move among Web pages rapidly because they understand the structure of Web site.

The level of motor ability was estimated from TappedBackButtonsCount with 73.91 % accuracy. The level of motor ability is high if the count of back button taps per second is low. Users who have high motor ability tapped the back button less often because they tend to make correct selections.

The models of the other skills could also estimate skill level with high accuracy, but more analysis is needed to interpret the structure.

Fig. 3.
figure 3

Estimation models of 6 skills

Full size image

5 Conclusion

To build estimation models of the key skills of the elderly for accessing online shopping Web sites, we conducted experiments and gathered operation logs. We applied machine learning to the logs to build decision tree-based estimation models of 6 skills. Test results show that our models could very accurately estimate skill level; for example, the level of experience was estimated with 86.95 % accuracy from the count of unique accessed Web pages per second. Seeking other effective feature values is needed to optimize the models, and further experiments are required to validate the models’ performance in other situations.