Creation of a Short Version of the User Experience Questionnaire UEQ

1 Introduction

The construct “user experience” (UX) refers to “all the users’ emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviors and accomplishments that occur before, during and after use” [6]. According to UX, the user forms a subjective and comprehensive opinion based on his experiences with interactive products and services [25].

This paper was written in cooperation with a company of the home appliance industry which has incorporated the achievement of a positive UX as an integral part in its product development process. The company decided to use the User Experience Questionnaire (UEQ; [18]) for regularly measuring UX since it evaluated the questionnaire as a suitable quantitative measuring instrument for its specific products – home appliances. Unlike e. g. the questionnaire AttrakDiff [9], the UEQ is constantly being further developed and adapted by its authors and additionally offers a regularly monitored benchmark tool. Furthermore, a large variety of language versions of the questionnaire is available.

However, especially in the context of user tests with repeated measurements or in long-term studies, the UEQ entails some drawbacks. Fatigue effects may occur and affect the data quality, especially when participants have to use the UEQ repeatedly during one test session to assess multiple products or prototypes. This occurred regularly at the mentioned consumer goods company. A short version of the UEQ can provide a greater degree of feasibility and reliability, since it facilitates quick and multiple use of the questionnaire during a test session. Thus, e. g. accidental ticking of responses due to fatigue can be prevented. Furthermore, through the multiple application of the questionnaire an evolution of the user’s perception process can be shown during the test [10].

The authors of the UEQ advise against eliminating single items, as this weakens the reliability due to the items’ interdependence [24]. The objective of this paper is to reduce the questionnaires’ items while maintaining high internal consistency and validity of the measuring instrument.

2 The User Experience Questionnaire (UEQ)

The User Experience Questionnaire (UEQ; [17]) is based on the model of hedonic and pragmatic quality perception [11]. The UX is thus defined as a construct consisting of the user’s perceived pragmatic (PQ) and hedonic quality (HQ) as well as of a general evaluation of the product. Both mentioned qualities are evaluated independently. It has been shown that the two quality factors PQ and HQ are only weakly correlated [17], and that they both have a direct effect on the global valence dimension Attractiveness (ATT). This last dimension describes the general acceptance or rejection of the product or service [24]. The data collected through the UEQ do not provide any information about the reason for rejection or approval of the respective product or service [16].

The UEQ consists of 26 items in the form of semantic differentials. These items are grouped into six scales: Attractiveness (ATT), Efficiency, Dependability, Perspicuity, Stimulation, and Novelty. Efficiency, Perspicuity and Dependability are part of the task-related or Pragmatic Quality (PQ). Stimulation and Novelty are part of the non-task-related or Hedonic Quality (HQ) (see Fig. 1; [17]). The Efficiency scale refers to the degree of efficient and fast task processing attainable with the help of the product or service. The Perspicuity scale evaluates whether the user can immediately trust in the product or service and if it enables straightforward access. Dependability refers to the degree of the user’s control over the product or service. The Stimulation scale will be highly ranked if the user experiences the product or service as being interesting and exciting and is strongly motivated to use it. Novelty describes to which degree the product or service is perceived as innovative or original by the user. Attractiveness (ATT) refers to the general approval or rejection of the product or service [24].

Each scale comprises four highly correlated items, except for the ATT scale with six items. The authors emphasize the enhanced robustness achieved through the specific number of items [22]. The polarity of the adjective pairs is random. The items can score values between +3 and −3 [17].

A data analytic approach was used when creating the UEQ [5]. First, pairs of adjectives were collected in brainstorming sessions with usability experts. 153 test persons were then asked to evaluate interactive products using these adjective pairs. The dimensions and scales of the UEQ were defined through exploratory factor analyzes. For the items of the dimension ATT, a separate factor analysis was conducted. The items with the strongest factor loadings were selected for the final version of the User Experience Questionnaire [17].

The validity of tools for UX measurements is usually examined by measuring correlations between the dimensions and scales of UX and various aspects of user satisfaction. Possible aspects of user satisfaction are the general contentment when interacting with a product or service, the intention to use a product or service itself, or specifically recommending it. Satisfaction arises when the expectations made of an interaction and the actual experiences match. Therefore, it can be equated with a global assessment process [15]. Hassenzahl et al. [10] mention the correlation between a high level of satisfaction and a positive UX. Bevan [1] describes satisfaction as a general consequence of the achievement of hedonic and pragmatic goals or a good UX.

Since 2009 there has been a short version of the UEQ known as “Short UEQ”. However, this version still contains 20 items and is therefore only marginally shorter than the long version. As part of the reduction, the Dependability scale and two items on the ATT scale were eliminated as they achieved the weakest values in terms of internal consistency and factor loadings [23]. The authors of the UEQ recommend eliminating an entire scale rather than a single item. When a single item is eliminated, reliability is weakened due to the interdependence of the scales. This makes comparability with other products or services more difficult [24].

Figure 1

Dimensions and scales of the UEQ [21].

The objective of this paper is to reduce the UEQ by about half of its items without drastically impairing the reliability and validity values of the questionnaire. The reduction is based on a data set from the said home appliance company. The practical relevance of a short version of the UEQ arises from the need to use the UEQ during a test situation repeatedly to assess the interaction with multiple stimuli, while maintaining comparability with “normal” UEQ studies.

3 Method

4 Results

As a result of the described analysis process, a short version of the UEQ was created with eleven items that fulfill all postulated requirements and can thus be regarded as reliable and valid. Tab. 1 gives an overview of the short version of the UEQ, including an assignment of the eleven items to the three dimensions ATT, PQ and HQ.

After exploratory factor analyses on the six items of the ATT dimension, the two items “attractive – unattractive” and “friendly – unfriendly” were deleted. The scale still contains four items after the reduction. The communalities of all remaining items of the ATT scale are greater than .70, and the extracted factor explains 74.5 % of the variance of the four items. All factor loads are greater than .80.

As a result of the exploratory factor analyses of the 20 items of the dimensions PQ and HQ, 13 items were eliminated (“not interesting – interesting”, “unpredictable – predictable”, “fast – slow”, “usual – leading edge”, “secure – not secure”, “meets expectations – does not meet expectations”, “impractical – practical”, “organized – cluttered”, “conservative – innovative”, “easy to learn – difficult to learn”, “clear – confusing” and “boring – exciting”). Four of the seven remaining items load on the factor 1 (PQ) and three on the factor 2 (HQ). The communalities of all remaining items are greater than .65 and the extracted factors explain 73.4 % of the variance of the seven items. All items load on their respective factor with at least .80, except the item “inefficient – efficient”, which still has a very high factor load of .72. The factor loads are all unambiguous, since each item loads with less than .31 on a second factor.

The Cronbach’s alphas of the dimensions ATT, PQ and HQ in the short version are all above the value of .80 and attest to high reliability of the questionnaire. A comparison of the internal consistencies between the long and the short version of the UEQ shows that the internal consistency of the short version has slightly declined but still achieves a very good value (see Tab. 2).

A confirmatory factor analysis on the short versions’ dimensions indicates very good results (see Fig. 2). The standardized coefficients and R² achieve values greater than .60 (except item 13: R² = .49). The fit indices of the model range from good to acceptable (RMSEA = .065, CFI = .976, SRMR = .031).

In terms of criterion validity, the short version of the UEQ achieves good values. However, these values do not approach the level of the long version. The correlation values of the dimensions of the shortened questionnaire version with the three general satisfaction items vary between .34 and .51 (p < .01). By comparison, the correlation values of the long UEQ version are slightly higher, ranging from r = .41 to .53 (p < .01) (see Tab. 3).

Figure 2

Results of a confirmatory factor analysis with the three dimensions of the short version of the UEQ.

The short version also shows good results with respect to the other postulates: The dimension ATT correlates most strongly of all three dimensions with the six items about general satisfaction. Postulate (1) is thus fulfilled. The dimension HQ achieves better values in both tests than the dimension PQ, which leads to the rejection of the second postulate (2). The third postulate (3) is only partially fulfilled: The two dimensions PQ and HQ in both questionnaire versions highly correlate with the dimension ATT (r between .71 and .88, p < .01). However, the dimensions PQ and HQ also strongly correlate with one another, but less so in the short version than in the long version (UEQ long: r = .77, p < .01, UEQ short: r = .56, p < .01; see Tab. 3).

Postulates (4), (5), and (6) are fulfilled: In both the short and long version of the UEQ, the HQ dimension correlates more strongly with “design” and “innovative features” than the dimension PQ. The item regarding usability correlates more strongly with the dimension HQ (see Tab. 4).

5 Discussion

The short version of the UEQ yields a reliable and valid measuring instrument. The goal of this study is thus achieved. The number of items is reduced by more than 50 %, permitting efficient and effective use of the measuring instrument in practice. The factor loads of all items achieve values higher than .80 and can thus be evaluated as very high (the only exception being the “inefficient – efficient” item, which still has a high factor load of .72). The theoretical model of the UEQ is embodied in the short version, since each scale is still represented by at least one item. Despite a reduced number of items, the internal consistency is still very high. The reliability of the short version differs only slightly from the original version. The problem of eliminating individual items as stated by Schrepp [24] was not confirmed.

The criterion validity of the short version of the UEQ can be rated as satisfactory. As expected, the correlation values of the three dimensions of the shortened UEQ with the items “user satisfaction”, “recommendation” and “re-purchase” are all significant and range from a middle to high level. However, all values of the short version are slightly lower than the same correlations in the original version of the UEQ. Four out of six additional postulates can be confirmed according to the data. These postulates describe the connection of the three dimensions with each other and with the general and specific satisfaction items. Contrary to expectations, the HQ dimension achieves better values than the PQ dimension, both in the short and in the original version of the UEQ. A possible explanation could be the two-factor theory according to Herzberg et al. [13]. Thus, a good PQ protects against dissatisfaction on the part of the user. However, once a base threshold of PQ has been reached, further enhancing this quality may not result in an equal increment in user satisfaction. The HQ is therefore the motivator. If positive values are achieved, the satisfaction values are high. However, if the HQ values are rather low, this does not automatically mean that the user is dissatisfied [10]. Furthermore, contrary to Laugwitz et al. [17], a strong correlation between the two dimensions PQ and HQ is found in the data of the long and short version of the UEQ. However, both dimensions also correlate positively with the dimension ATT, as postulated by the model.

While these are encouraging results, it should be noted that the satisfaction items used for validation purposes in this study only represent individual, very global statements and do not fully reflect all intricacies of the satisfaction construct. Therefore, it would be appropriate to conduct further studies using an empirical measurement tool for satisfaction in the form of a more detailed questionnaire. Furthermore, it might be useful to test the construct validity of the shortened UEQ by calculating the correlation values with a construct-related test [3]. This would be conceivable e. g. by calculating correlations between the UEQ and the AttrakDiff2. The UEQ was constructed based on this questionnaire, and the AttrakDiff2 already served for validation of the UEQ [16]. The validity of our findings is further constrained by the fact that the data set used is based on a specific product category, namely home appliances. Therefore, the extent to which our results can be generalized to cover other products and services is limited. Another critical aspect of the data set is that the test persons have been using the products for a longer period of time. Furthermore, the product categories tested in the dataset of this study varied greatly, and the participants were not required to complete a specific task with their products, but only to rate their experience from memory. Generally, users mainly remember the experiences they have only recently perceived. Thus, it may be possible to get a distorted assessment of the user experience [12]. Hence, it could be useful to replicate the findings of this study with another dataset where participants were instructed to complete specific tasks within a single product category.

For practical purposes when applying the shortened UEQ, it would be useful to adjust the UEQ analysis tool accordingly, so as to make a quick evaluation after user tests possible. In addition, it is recommended that the order of the items of the shortened UEQ version be kept in line with the full version in order to ensure randomness.

On a final note, one advantage of the shortened UEQ as presented is that it still represents the model on which the original UEQ (see Figure 1) is based. However, this is true only at the dimension level and not at the scale level. Three scales (Novelty, Efficiency and Dependability) in the short version of the UEQ contain only one item each and are therefore displayed incompletely and should be interpreted with caution. This was deemed acceptable, since the goal of this study was to create a new version of the UEQ that is as short as possible for multiple test situations. If an interpretation of the UEQ on the scale level is of immediate necessity, three further items should be included in the shortened questionnaire, so that each scale is represented with at least two items. Based on our analysis, these items should be “conservative – innovative” (item 26) for the Novelty scale, “organized – cluttered” (item 23) for the Efficiency scale and “meets expectations – does not meet expectations” (item 19) for the Dependability scale, since these items achieved the highest factor loadings in our study.

6 Conclusions

This paper presents a short version of the UEQ. A research gap can be closed in this area: Although there is already a slightly shorter version of the questionnaire, the “Short UEQ”, it differs only insignificantly in length from the original measuring instrument. Furthermore, one scale was omitted completely in the “Short UEQ”. In the short version now created, all scales of the original version are still represented by one or more items and thus give a comprehensive evaluation of the UX construct. In addition, the modified questionnaire is 50 % shorter than the original. The practical benefit of this reduction is to enable more efficient user testing. It also facilitates collection of multiple data from a participant within a single test. The new, shorter UEQ questionnaire provides a leaner yet still valid and reliable instrument for measuring UX.

Furthermore, possible problematic aspects and influencing factors are pointed out and the necessity for follow-up studies is stressed. It should be emphasized that the universal validity of the data must be further examined.