Keywords

1 Introduction

Currently, two major characteristics are shaping modern society. The first characteristic refers to the profound demographic change that is marked by an increasingly aging population. The population of senior adults is getting larger. The second characteristic is the ongoing proliferation of information technology products, such as web pages, in many parts of daily life [1]. The Internet is no longer inaccessible for older adults and they can also enjoy the convenience of the Internet. Despite the various appearances of Internet resources, seniors still have some disadvantages in effectively using Internet resources such as web pages. A previous study has shown that older people face more difficulties than younger people finding information on a website [2]. Meanwhile, a survey has found that a substantial portion of the senior population will never join the Internet community due to disability or the difficulty of using a web page [3]. This phenomenon has led to a number of studies. In particular, the mental model has been widely used to explain this phenomenon and it has been applied to human-computer interaction (HCI).

Many researchers have used the mental model to explore user performance. Some researchers have found that the mental model was positively correlated with user performance [1, 4,5,6] while others have found that the mental model either has an adverse effect or no significant effect on user performance [7, 8]. There may be two explanations for why these researchers have different and even conflicting results. The first is the method of eliciting mental models. The traditional method to elicit mental models of hierarchical systems is card sorting [9]. However, card sorting cannot be used to elicit the mental model of information structure considering the directional relationship, which is a key element of information structure. The second is that most researchers have explored the effects of mental models qualitatively.

In a previous study, Schmettow and Sommerr [7] had proposed an index called mental model similarity to investigate the effect of the mental model through a quantitative perspective and found that the match between mental model and website structure has no effect on browsing performance. However, Huang et al. [10] found that information structure had a significant effect on older adults’ performance and the mental models of older adults were positively correlated with user performance. Another previous study has found that users will have navigational issues when the information structure does not match the users’ mental model [11]. The reason why these results are different may be the method used to elicit mental models, which was card sorting. This method failed to be used to elicit mental models of information structures with the directional relationship. Getting a better understanding of the impact of mental model similarity on user performance is important for academics and practitioners alike.

This study aims to investigate the effect of mental model similarity on older adults’ performance as well as the age effect. Given that researchers have already investigated the mental model similarity effect on younger adults [12], this study mainly intends to investigate whether mental model similarity has an effect on the performance of older adults. The methods which were used to elicit mental models were card sorting and path diagram. The path diagram can be used to elicit mental models of information structures with the directional relationship, which had been verified in previous work. Specifically, in the experiment the older participants are asked to navigate three web pages built with a net structure, tree structure, and linear structure. The results of this study may give researchers an insight into the extent to which mental models affect performance.

2 Literature Review

2.1 Mental Model and User’s Performance

Since it was applied to HCI, the mental model has been widely used to analyze user performance. Studies on mental models can be approximately divided into two branches. The first branch is qualitative and the second branch is quantitative. There have been many qualitative studies on mental models. Some researchers have found that the mental model has a positive effect on the user’s performance. For example, Ziefle and Bay [4] found that users had better performance using the devices when they had better mental models of navigations menus. Slone [5] pointed out that the user’s website performance was affected by their mental model. Converse et al. [6] found that shared mental model was related positively to Team Decision Making (TDM). Huang.et al. [10] found that information structure had a significant effect on older adults’ performance and that mental models of older adults were positively correlated with user’s performance. Previous studies have also found that the users will have navigational issues when the information structure does not match their mental model [11]. However, some researchers have found that mental models have no significant effect on user performance. For example, Borgman [8] found that a mental model made no difference to the user’s performance of routine tasks. Furthermore, Payne [13] found that users who had the wrong mental model were still able to use the device well.

Compared with the qualitative studies, there are far fewer quantitative studies of mental models. The existing studies have focused on the match degree between the user’s mental model and website information structures. Schmettow and Sommer [7] introduced an index that they called mental model similarity to investigate whether or not the match degree between the mental model and website structure had an effect on user’s browsing performance. They found that there was no significant difference between mental model similarity and performance. However, researchers have already investigated the effect of mental model similarity on younger adults [12] and found that mental model similarity had a significant effect on the user’s performance.

2.2 Methods of Eliciting Mental Models

Many methods have been used to elicit mental models. Among them, card sorting is the most widely used method of eliciting mental models of hierarchical systems [4]. However, card sorting is not effective for eliciting mental models of information structure when the directional relationship among various elements of information structure is considered. In the previous work, researchers have proposed a path diagram to elicit the mental model [12]. This method adds the directional relationship based on the card sorting and it can be used to describe the understanding of a hierarchical system.

2.3 Age Difference

Age has been studied as a factor affecting use of new technologies and many studies have investigated the effect of age. For example, Phang et al. [14] found that older adults who perceived themselves in declining physical condition reported that e-government services are less easy to use. Zaphiris [15] has found that older adults have particular difficulties with deep menu structures and they tend to get more easily lost in broad or deep menus than young people. Ziefle and Bay [4] demonstrated that younger participants who had a fairly correct mental model have better performance than older participants, none of whom had a correct mental representation of the information structures. Kolodinsky [16] pointed out that the reasons why older people tend to shy away from the Internet is the lack of understanding from web developers that older people have different needs from their younger counterparts, both in the reasons for which older people use the Internet and also in the way that older users interact with it (mostly due to ageing-related functional impairments).

However, there have not been enough studies of age. In one of the few examples, Arning and Ziefle [1] pointed out that the factor ‘chronological age’ cannot be used as an explaining variable itself when it comes to usability research and design for older users. Consequently, this study aimed at a detailed examination of the age effect on mental model similarity.

3 Methodology

An experiment was conducted to investigate the effect of mental model similarity on older adults’ performance. Mental model similarity was calculated from card sorting and the path diagram using the mathematical formula that has been verified in the previous work [12].

3.1 Equipment and Materials

A notebook computer with a touch screen (ThinkPad YogaS1) was used. A Morae Recorder was used to present the task specification for the participants. Cards with the name of the nodes of three websites were used to place card sorting and a whiteboard was used to draw path diagrams. A phone was used to record the card sorting and the path diagram for analysis.

Web pages with net, tree and line structures were used. To avoid the learning effect, the knowledge about ancient inventions, books, and historical figures were used as the content of the websites and this knowledge was uncommon to see in daily life. The users’ spatial ability was tested by a KJ-I spatial location memory span tester.

3.2 Participants

A total of 14 older adults from urban and rural areas of Chongqing, China were recruited as participants. Older adults who were literate and aged above 60 were eligible for this study. The age of the participants ranged from 62 to 85 years old (Mean = 71.9, SD = 7.42). In total, there were nine male participants and five female participants. The experience of using computers, tablets, and smartphones (see Table 1) was investigated.

Table 1. The participants’ experience of using technology products (hours per day)
Full size table

3.3 Task

There were six tasks for each website and the tasks were focused on finding an item hidden in the web page. The participants firstly watched the task specification. They then found the target according to the task specification by touching the hyperlink in the web page. Finally, they answered two questions according to the content, which were used to check whether they had read the content carefully.

3.4 Dependent Variables

There are two dependent variables: task completion time and the number of clicks. The task completion time is the average value of the completion time of six tasks under each information structure. The number of clicks is the average value of the number of clicks to complete the six tasks under each information structure. Both the task completion time and the number of clicks were recorded by Morae Recorder.

3.5 Independent Variables

There are three independent variables: website information structure, mental model similarity between older adults and designers, and age. The mental model similarity was calculated from the card sorting and the path diagram using the mathematical formula, which has been verified in previous work [12]. Specifically, the directionless similarity was calculated from card sorting, and the directional similarity was calculated from the path diagram. The website information structure has three levels: net, tree and linear, which are commonly seen in daily life.

Age here was regarded as one independent variable and this study investigated the effect of age on mental model similarity through a quantitative perspective.

The demographic variables included technology product experience and spatial ability. The technology product experience was measured through a questionnaire. The participants’ spatial ability was tested through the spatial location-memory span tester. To avoid memory and learning effects, the order of the task was randomized.

3.6 Procedure

The experiment took each participant about 90 min. First, each participant filled out a consent form and a general questionnaire about his/her demographic information and experience with technology products. Second, a spatial ability test was conducted by using a spatial location–memory span tester. Third, training about how to place card sorting and draw path diagram was conducted. The training only ended when the participants got a full understanding of how to place card sorting and draw the path diagram. Finally, the participants completed tasks on each web page. After completing all six tasks of each page, the participants were required to place card sorting and draw a path diagram on a whiteboard. During the whole process, the participants were encouraged to complete the tasks individually. They could ask for help but it would be recorded.

3.7 Quantifying Mental Model Similarity

The method of quantifying mental model similarity has been well elaborated in previous work [12]. The method consists of two parts: the first part is used to calculate the directionless similarity and the second part is used to calculate the directional similarity. The formulae that are used here are drawn from Xie and Zhou [12]. The directionless similarity measure aij can be obtained using:

$$ a^{ij} = \frac{e^{ij} }{e^{ij} + b^{ij} + b^{ji} }. $$
(1)

Equation (1) was used to calculate the directionless similarity through card sorting. The directionless similarity can be calculated by calculating the number of identical and different elements of the information structure between two different card sortings.

The directional similarity measure rij can be obtained using the following formulae:

$$ c^{ij} = \sum\nolimits_{k} {\sum\nolimits_{l} {\hbox{min} \left\{ {h_{kl}^{i} ,h_{kl}^{j} } \right\}} } $$
(2)
$$ d^{ij} = \sum\nolimits_{k} {\sum\nolimits_{l} {|h_{kl}^{i} - h_{kl}^{j} |} } $$
(3)
$$ r^{ij} = \frac{{c^{ij} }}{{c^{ij} + d^{ij} }} $$
(4)

The adjacency matrix was used to describe the path diagram. Equation (2) was used to calculate the identical parts of two different path diagrams (mainly the directed segments in path diagram) and Eq. (3) was used to calculate the different parts of two different path diagrams. Equation (4) was used to calculate the directional similarity. These formulae were used to quantify the match degree between the user’s mental model and the information structure. See previous work [12] for a detailed description of these formulae.

4 Results and Discussion

4.1 Descriptive Statistics

There were a total of 14 participants in this experiment (nine males, five females). About 43% of the participants used multiple technology products—such as a computer or smartphone—and the rest had no experience of these products.

4.2 Information Structure and Mental Model Similarity

The directional similarity and directionless similarity were calculated using the formulae. The descriptive analysis results of directional similarity and directionless similarity are shown in Table 2.

Table 2. The descriptive analysis results of directional similarity and directionless similarity
Full size table

Table 2 indicates that the mean of directional similarity was lower than the mean of directionless similarity. One possible reason for this is that card sorting did not take the directional relationship of information structure into account, which makes it easier for users to elicit mental models. Once more details such as directional relationship are considered, the mental model similarity is changed. This verified that it is incorrect to elicit mental models without considering the directional relationship of the information structure.

Repeated variance analysis was conducted. The dependent variables were directional similarity and directionless similarity, the independent variable was information structure. The results of repeated variance analysis are shown in Table 3.

Table 3. The influence of information structure on mental model similarity
Full size table

A regression analysis was then conducted. There were three independent variables: information structure, technology product experience, and spatial ability. The dependent variables were directional similarity and directionless similarity. The results of the regression analysis are shown in Table 4.

Table 4. Regression analysis results of information structure on mental model similarity
Full size table

Table 4 indicates that the information structure had a marginal effect on directional similarity while it had no effect on directionless similarity. One possible reason for this is that the path diagram is more effective than card sorting when researchers elicited mental models of information structure. In addition, the number of subjects here was only 14, which is small. Different results may be found after expanding the number of subjects.

4.3 Mental Model Similarity and User Performance

A linear regression was conducted. The dependent variables were the task completion time and the number of clicks. The independent variables were directional similarity and the directionless similarity. The demographic variables were spatial ability and technology product experience. The results of the regression are shown in Table 5.

Table 5. The influence of mental model similarity on user’s performance
Full size table

According to Table 5, the directional similarity was marginally linearly related to the task completion time. The participants took less time when the directional similarity was higher. The directional similarity’s mean under net structure (Mean=0.19, SD=0.15) was lower than the tree (Mean=0.28, SD=0.25) and the linear structure (Mean=0.35, SD=0.31), which means that it took less time to complete a task in the net-structure web page than the other two web pages. These results are in agreement with those of Huang [10].

The effect was marginal and one possible reason is that the information structure had a significant effect on the user’s performance. Repeated ANOVA was conducted to investigate the effect of information structure on user’s performance. The dependent variables were the task completion time and the number of clicks, and the independent variable was information structure. The results indicated that the information structure had a significant effect on the task completion time and it had no impact on the number of clicks. One possible reason for this is that the older participants tended to focus on the content of the web page, and they would have more meaningless clicks to find the target item. The results of the repeated ANOVA are shown in Table 6.

Table 6. The effect of information structure on the user’s performance
Full size table

4.4 Age Effect

To investigate the influence of age on mental model similarity, one-way ANOVA was conducted. The age and the mental model similarity were categorized by the average. The dependent variables were directional similarity and directionless similarity. The independent variables was age. The results of one-way variance analysis are shown in Table 7.

Table 7. The influence of age on mental model similarity
Full size table

Table 7 shows that age had significant influence on mental model similarity. The extent to the effect of age on mental model similarity was investigated and the results are shown in Table 8. Table 8 indicates that age had different degrees of influence on mental model similarity. Younger adults had higher mental model similarity than older adults, showing that younger adults had a better understanding of information structure than the older adults. One possible reason for this is that younger adults have more experience using technological products such as a computer or smartphone. The older participants who had enough experience of technology products acted as well in the test as the younger adults. Table 8 also indicates that the effect of age on directional similarity is larger than the effect of age on directionless similarity. One possible reason is that drawing path diagram is more difficult for older adults.

Table 8. The difference of mental model similarity between younger and older adults
Full size table

The influence of age on user’s performance was investigated. The results of one-way ANOVA showed that age had a significant effect on the task completion time (F (1,132) = 103.34, p = 0.00) and the number of clicks (F (1,132) = 4.25, p = 0.41). The extent to the effect of age on user’s performance was investigated and the results are shown in Table 9.

Table 9. The difference of user’s performance between younger and older adults
Full size table

Table 9 indicates that younger adults took less time and clicks that older adults. In addition, the effect of age on the task completion time is larger than the effect of age on the number of clicks. One possible reason is that older adults had to take more time to read the content in the experiment.

5 Discussion

First, the method which was proposed to elicit the mental model in previous research [12] is effective for eliciting the mental model of older adults. The results indicated that the information structure had marginal effects on directional similarity while it had no effect on directionless similarity. This is a little bit different from the results of a previous study which found that the information structure had a significant effect on directional similarity [12]. One possible reason for this is that there were 14 older participants in the present study, which is smaller than the number of younger adults. Hence, expanding the number of older adults may give different results.

Second, the directional similarity had a marginal impact on the task completion time while it had no impact on the number of clicks. One possible reason for this is that in this experiment the older participants tended to focus on the content of the web page while always ignoring the relationship between each page. Thus, the older participants would have more meaningless clicks to find the target item while the younger adults would not. Another possible reason is that the information structure had a significant effect on the user’s performance. The results of repeated ANOVA indicated that the information structure had a significant effect on the task completion time while it had no impact on the number of clicks.

Third, an age effect was observed. The results showed that age had a significant effect on mental model similarity and user’s performance. Younger adults had higher mental model similarity than the older adults. In addition, younger adults took less time and clicks to complete tasks than the older adults.

6 Conclusion

This study investigated the effect of mental model similarity on the older adults’ performance, as well as the age effect. There were three main findings:

First, information structure had a marginal impact on the directional similarity while it had no significant effect on directionless similarity. This is almost same as the previous finding of younger adults, which indicated that the information structure had a significant effect on the directional similarity while it had no significant effect on directionless similarity.

Second, the directional similarity had a marginal effect on the task completion time while it had no impact on the number of clicks. Directionless similarity had no effect on the task completion time or the number of clicks.

Third, an age effect was observed. Younger adults had higher directional similarity and directionless similarity than older adults. Younger adults also took less time and clicks to complete tasks than the older adults.

Future work should be conducted in order to: (i) increase the number of older participants in order to get a full understanding of the mental model similarity’s impact on; and (ii) expand the types of tasks and information structure.