An empirical evaluation of the User Engagement Scale (UES) in online news environments

Highlights

• We examined the robustness of the User Engagement Scale (UES).

• Three studies were conducted in Canada and the United Kingdom.

• The UES sub-scales were reliable across three samples of online news browser.

• A four-factor structure, rather than six, may be more appropriate.

• The UES differentiated between online news sources and experimental conditions.

Abstract

Questionnaires are commonly used to measure attitudes toward systems and perceptions of search experiences. Whilst the face validity of such measures has been established through repeated use in information retrieval research, their reliability and wider validity are not typically examined; this threatens internal validity. The evaluation of self-report questionnaires is important not only for the internal validity of studies and, by extension, increased confidence in the results, but also for examining constructs of interest over time and across different domains and systems.

In this paper, we look at a specific questionnaire, the User Engagement Scale (UES), for its robustness as a measure. We describe three empirical studies conducted in the online news domain and investigate the reliability and validity of the UES. Our results demonstrate good reliability of the UES sub-scales; however, we argue that a four-factor structure may be more appropriate than the original six-factor structure proposed in earlier work. In addition, we found evidence to suggest that the UES can differentiate between systems (in this case, online news sources) and experimental conditions (i.e., the type of media used to present online content).

Introduction

In information interaction research, there is growing recognition for information systems to satisfy both utilitarian and hedonic needs, particularly in the area of exploratory search where users’ motivations go beyond merely completing an information task (White & Roth, 2009). This has created a call to action for researchers to design systems that are fun and engaging and that foster learning and discovery (White & Roth, 2009), but also to systematically consider the measurement of such experiences. Kelly (2009) underscores the need to capture affective responses during “particular IR [information retrieval] interactions and states” (p. 200), while O’Brien and Lebow (2013) advocate developing robust measures of experience. There is an increasing impetus to support searchers to not only retrieve and evaluate information, but to engage with it. Further, we need “good measures” that capture searchers’ affective experiences and allow a degree of standardization to facilitate comparisons between different search systems, user groups, etc. (Käki & Aula, 2008, p. 86).

Questionnaires are commonly used to measure attitudes toward systems and perceptions of search experiences. Kelly (2009) notes that such measures have good face validity, but are often developed “ad hoc” and their reliability and wider validity are not well established. This is problematic because, without reliable and valid measures, how can we be confident of our findings (Cairns, 2013)? Thus it is essential to evaluate the robustness of self-report measures to bolster internal validity.

This paper looks to a particular questionnaire, the User Engagement Scale (UES) to evaluate its potential as a “good measure” for adequately assessing human information experiences. The UES is a 31-item self-report questionnaire developed in the e-shopping domain (O’Brien & Toms, 2010a). It has been subject to some evaluation in search systems (Arguello et al., 2012, O’Brien and Toms, 2013), multimedia presentation software (O’Brien & Toms, 2010b), social networking applications (Banhawi & Mohamed Ali, 2011), and games (Wiebe, Lamb, Hardy, & Sharek, 2014) with mixed results regarding its generalizability in experimental settings, and the number of user engagement dimensions (UE) is captures. Given these mixed results, our motivating question was: “Is the UES a reliable and valid experiential scale to use in information interaction research, specifically online news browsing?” Here, we investigate its viability across three studies.