1 Introduction

Successful use of ICT requires domain knowledge and interaction knowledge. The basic interaction knowledge required for successful use of computers can be called “computer literacy”. It shapes and is shaped by the use of ICT and is less common among older adults.

This paper describes the validation of the computer literacy scale (CLS) introduced by [14], following five steps. First, the CLS and related measures computer expertise (CE), Computer Proficiency (CPQ), control beliefs regarding technology use (KUT), attitude toward technology (ATT) and computer anxiety (CATS) are briefly described. Second, convergent and discriminant validity is described using correlations and a principal component analysis (PCA). Third, criterion validity is described with predictions of successful ICT use exemplified with (1) the use of different data entry methods and (2) the use of different ticket vending machine (TVM) designs. Fourth, since CLS is work in progress, the current developmental status and an outlook on upcoming procedural knowledge items and an adaptive CLS are provided. Finally, readers are encouraged to use the CLS in their own research involving human computer interaction and to contribute to the continuous validation and improvement of the CLS.

2 Method

This paper reports results of validation studies that used CLS and other measures of computer related user characteristics to predict successful use of diverse ICT applications for young and old age groups in two independent applications, namely the use of ticket vending machines (TVM, N = 124, [13]) and different data entry methods (mouse, touch screen, eye gaze) and navigation in complex information spaces (N = 90, [8]). Convergent and discriminant validity of CLS will be reported along with psychometric properties and correlations with user characteristics such as computer expertise (CE), Computer Proficiency (CPQ), control beliefs regarding technology use (KUT), attitude toward technology (ATT) and computer anxiety (CATS).

2.1 The Computer Literacy Scale (CLS)

The CLS is an objective knowledge test of ICT-related symbols and terms commonly used in the graphical user interface of interactive computer technology. It has been designed specifically for older adults with little computer knowledge and is based on the idea that knowing common symbols and terms is as necessary for using computers, as it is for reading and writing letters and books.

The CLS focuses on a small but essential aspect of computer literacy and uses it as indicator for the broader construct: “If literacy can be considered the ability to read symbols and use them, then computer literacy could be considered the ability to understand and use computer related symbols, functional elements and interaction patterns” ([14], p. 8). These basic building blocks of computer literacy are tested in an objective knowledge test with 26 items (21 symbols and 5 terms) in a matching task, taking about 15 min to complete, depending on literacy level. The CLS can be downloaded for free as printable pdf. It is available in English, German and Spanish. Figure 1 shows sample items in the matching task.

Fig. 1.
figure 1

Sample items in the matching task of the CLS

Full size image

2.2 The Computer Expertise Questionnaire (CE)

Based on the INCOBI computer expertise inventory by [11, 1] introduced the computer expertise (CE) questionnaire with 18 items assessing theoretical (9 items) and practical (9 items) computer knowledge by describing typical tasks or problems that occur using computers and asking participants to mark the optimal course of action in a multiple choice task with 4 alternatives [1].

2.3 The “Computer Proficiency Questionnaire” (CPQ)

Later, [2] developed the “Computer Proficiency Questionnaire” (CPQ) to assess the computer proficiency of seniors (from non-users to frequent computer and Internet users), arguing that the CE by [1] had been developed with data from older adults who had substantial computer experience and that the CLS by [14] had been designed for older adults, but focused largely on declarative knowledge rather than the ability to perform computer tasks. The CPQ consists of 33 items (there is also a short form with 12 items) that ask the respondent whether she can use technology divided in six categories: computer basics, printer, communication, Internet, calendar and entertainment. Thus, the CPQ is not a knowledge test but a self report measure.

2.4 Control Beliefs for Interaction with Technology (KUT)

The KUT (German akronym for “control beliefs for interaction with technology”) was developed 1999 by [3] to extend the scope of observed user characteristics with a personality construct that guides user actions. Participants indicated on a Likert scale of 0 (not true at all) to 4 (absolutely true) their control beliefs regarding technology use, resulting in a maximum total score of (8 × 4 =) 32. A complete list of translated items, including polarity, mean, variance and discriminatory power can be found in [12].

2.5 Attitude Towards Technology (ATT)

Attitude determines motivation to use technology and thus will have a direct influence on successful use and an indirect influence through better experience and practice [6, 15]. Attitude towards ticket vending machines was measured using an eight item seven point semantic differential created for this study.

2.6 “Computer Anxiety Trait Scale” (CATS)

Computer anxiety and computer knowledge often show a strong negative correlation (e.g. [10], r = −.83, p < .01, N = 222). Anxiety towards ticket vending machines was measured using an adaptation of the “Computer Anxiety Trait Scale” (CATS) by [7], asking the participants to imagine being at the train station wanting to use a TVM and to rate their approval to 16 statements (e.g. “I sweat”, “My heart beats faster”) regarding this situation on a five-point Likert scale ranging from “not at all” to “absolutely”.

3 Results

Unless marked otherwise, results regarding TVM use are from a study described by [13] (N = 124) and those regarding different data entry methods (mouse, touch screen, eye gaze [9]) and navigation in complex information spaces from a study described by [8] (N = 90).

3.1 Reliability

When the CLS was introduced [14], the quality of the computer literacy scale was assessed with internal consistency, discrimination power and item difficulty measures. Internal consistency was high with a Cronbach’s alpha between .93 and .96, indicating high homogeneity, with discrimination power ranging from r = .22 to r = .84. Item difficulty was low for the young group, but reasonably broad for the old group, ranging from P = .13 to .87. Kolmogorov-Smirnov-Tests revealed, that the CLS scores were normally distributed for the old group (D(39) = 0.10; p > .10) but not for the young group (D(81) = 0.12; p < .01), for whom most items were too easy. Table 1 shows reliability (Cronbach’s alpha) and number of items for the scales CLS, KUT, ATT, CATS, CE and CPQ, indicating high reliability of the measures investigated.

3.2 Face Validity

Face validity of the CLS can be considered high, because the test directly asks for the meaning of terms and symbols associated with ICT use. Test participants did not doubt that knowing these would be relevant for successful ICT interaction. As with any objective knowledge test, there is little impact of personality traits when compared to subjective self report measures, which is generally perceived as an advantage.

Table 1. Scale reliability and number of items for the scales CLS, CE, CPQ, KUT, ATT and CATS.
Full size table

3.3 Construct Validity

3.3.1 Computer Experience and Expertise (Convergent Validity)

Computer experience is necessary but not sufficient for high computer literacy. To capture different aspects of computer experience, it was operationalized using three measures added on the first page of the CLS: duration (measured in years), intensity (measured in hours per week) and diversity (measured in frequency of use for different computer applications).

In [14] older adults reported a mean duration of 7 years using computers and a mean intensity of 3 h per week, while younger adults reported to have used computers for a mean duration of 10 years, which was not significantly longer than the older group, and to spend an average of 27 h per week using computers, which was significantly more than the older group (t(18.49) = −7.06, p < .01, r = .85). For diversity of computer use, the older group scored M = 4.14 points while the younger group scored M = 14.69 points (max = 21), constituting again a significant difference t(28) = −6.70, p < .01, r = 0,78). As expected, computer experience and computer literacy were highly correlated. See Table 2 for an overview. In order to assess the unique contribution of computer literacy and computer experience on TVM-performance, a partial correlation analysis was conducted. The best predictor of performance was computer literacy R2 = .37), followed by diversity of computer experience (R2 = .25).

Table 2. Computer experience measured in duration, intensity and diversity in the older and younger participant group and their correlation with the CLS score.
Full size table

As another measure of convergent validity, the relationship of the CLS and the computer expertise questionnaire (CE, [1]) was investigated. A total of n = 90 adults (M = 47.5 years, SD = 16.8, 36 female, 54 male) participated in a study conducted and described in detail by [8]. Results show moderate correlations between CLS and CE (τ = .62, p < .01), indicating that they measure related constructs, even though they do so in very different ways.

In another study [8], a paper folding and a cube rotation test were administered additionally to measure mental rotation ability, which is considered relevant to navigate virtual spaces. Correlations of these measures can be seen in Table 5, indicating strong relations between them and even stronger relations between CLS and CE (Table 3).

Table 3. Correlations for the scales CLS, CE, paper folding and the cube rotation test
Full size table

3.3.2 Discriminant Validity

It was expected that besides computer literacy (CLS), interaction with computers would also be related to other user characteristics such as control beliefs regarding the use of technology (KUT), attitude toward technology (ATT) and computer anxiety (CATS). As Table 4 shows, these user characteristics were indeed highly correlated with Pearson correlations between .46** and \xE2\x88\x92.73**.

To test whether the 58 items taken from these ICT use related scales actually measured distinct traits, a principal component analysis (PCA) with varimax rotation (with Kaiser normalization procedure) was conducted (total sample N = 124). The Kaiser-Meyer-Olkin measure confirmed sampling adequacy for the analysis with KMO = .82 and Bartlett’s test of sphericity χ2(1653) = 4643.59, p < .001, indicated that correlations between items were sufficiently large for PCA. Four factors that explained 48.6 % of variance in all items were extracted. The rotated factor matrix in table A1 (appendix) shows that all items loaded on the appropriate latent constructs (indicated by bold values), confirming the factorial validity of the different scales.

Table 4. Pearson correlations for usability measures and user characteristics in the control, video and wizard conditions
Full size table

3.4 Criterion Validity

A central question regarding the validity of the CLS is: How well does it predict actual success in ICT use? Success in TVM use was operationalized according to the usability criteria [5] effectiveness (in solving 11 tasks using the TVM to select tickets), efficiency (measured in the time and the steps needed to solve the tasks) and satisfaction (measured as the mean score of 13 items based on the Questionnaire for User Interface Satisfaction (QUIS) by [4]). ICT use was operationalized using three different TVM designs: (1) a simulation of the original TVM as used by the Berlin Public Transport System (2) the same TVM with a brief (2:37 min) instructional video before use and (3) a wizard redesign of the TVM that maintained the same functionality but was designed to require less computer literacy to be universally usable (see [13]) for details).

To estimate the impact of the user characteristics on usability measures in the original TVM, video and wizard conditions, first separate Pearson correlations are reported (see Table 4).

Using the original TVM, age was strongly related to effectiveness and efficiency but not to satisfaction. The same is true for the video condition, showing very similar if a little weaker negative correlations. The wizard redesign however shows no significant correlations between age and effectiveness, efficiency (steps) or satisfaction, indicating successful inclusive design - only the correlation to efficiency (time) persists. Interestingly, this pattern is largely replicated with the other user characteristics. Table 5 shows correlations between usability measures and user characteristics over all TVM designs, indicating that CLS has the strongest correlations with all usability measures.

Table 5. Pearson correlations (above the diagonal) and significance values (below the diagonal) for usability measures (N = 124, except for satisfaction N = 118) and user characteristics (N = 102, except for satisfaction N = 98)
Full size table

Finally, a hierachical multiple regression with blockwise entry of (1) age (2) CLS and (3) KUT, ATT, CATS was conducted to estimate the degree to which these user characteristics predict effectiveness, efficiency and satisfaction, of which only effectiveness shall be reported here. As results presented in Table 6 show, age was a strong predictor of effective use of the original TVM with a β = −.54**. Yet if the CLS score was entered into the regression model, the impact of age was reduced to a non-significant β = −.17 and CLS was the best predictor (with β = .48m). CLS remained the best predictor after KUT, ATT and CATS had been added in a third step. For the video and the wizard condition, the impact of these user characteristics was reduced, indicating that the design changes lowered some barriers to successful TVM use. Thus especially the wizard came close to the goal of universal usability. See Table 6 for an overview of the results of the hierarchical regression.

In another study [8] CLS and CE were administered before participants used a project planning software with four different layouts (control layout, Overview window, detail window and zoom function). In Table 7 the correlations between CLS, CE and the number of mistakes are presented. Table 8 described the correlations between CLS, CE and total execution time, participants needed to solve the tasks with the help of the different layouts. As you can see in Table 7, both, CLS and CE predict the number of mistakes. Regarding the execution time interesting results could be found (Table 8). In this case, only the CLS scale predicts total execution time.

4 Discussion

It is argued that ultimately, computer literacy could have an impact on any ICT-interaction and thus it should be measured as control variable in any study using ICT.

The CLS has shown to be a reliable and valid measure of computer literacy. It can be used to assess the computer literacy of a person as well as the computer literacy requirements of a user interface design.

Since symbols and terms used in human computer interaction change quickly, it remains a constant challenge to update and improve the CLS. From the beginning of the CLS development it was a central goal to improve computer literacy assessment with an objective knowledge test rather than subjective self report measure. With the ongoing development of the CLS, the test of declarative knowledge described above has been extended with a procedural knowledge test, in which users are asked to complete tasks online that require computer literacy [16]. Figure 2 shows screenshots of two of these tasks, booking a flight (left) and mixing colors (right).

Table 6. Effects of age, CLS, KUT, ATT and CATS on effectiveness for original TVM, video and wizard re-design
Full size table
Table 7. Correlations for the scales CLS and CE with the number of mistakes in the four investigated layouts.
Full size table
Table 8. Correlations for the scales CLS and CE with the total execution time in the four investigated layouts.
Full size table

Currently, efforts are directed at making an adaptive CLS available online, which will reduce testing time to about 3 min, without compromising reliability or validity of the results. The item base has been extended to over 120 items that have been tested and selected for conformity to the RASCH model, providing a solid base for adaptive testing. Figure 3 shows screenshots with sample items from the current prototype of the adaptive CLS. The adaptive CLS will be available shortly at www.computer-literacy.net. We invite everyone to use it in their own studies and would appreciate any feedback regarding the CLS that helps us to continually improve it.

Fig. 2.
figure 2

Screenshots of two tasks for the extended CLS, designed to test procedural computer interaction knowledge. The task on the left is to book a flight, on the right colors are to be chosen to mix a paint.

Full size image
Fig. 3.
figure 3

Screenshots with sample items from the current prototype of the adaptive CLS

Full size image