On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state

Abstract

This paper reports results from a controlled experiment (N = 50) measuring effects of interruption on task completion time, error rate, annoyance, and anxiety. The experiment used a sample of primary and peripheral tasks representative of those often performed by users. Our experiment differs from prior interruption experiments because it measures effects of interrupting a user’s tasks along both performance and affective dimensions and controls for task workload by manipulating only the time at which peripheral tasks were displayed – between vs. during the execution of primary tasks. Results show that when peripheral tasks interrupt the execution of primary tasks, users require from 3% to 27% more time to complete the tasks, commit twice the number of errors across tasks, experience from 31% to 106% more annoyance, and experience twice the increase in anxiety than when those same peripheral tasks are presented at the boundary between primary tasks. An important implication of our work is that attention-aware systems could mitigate effects of interruption by deferring presentation of peripheral information until coarse boundaries are reached during task execution. As our results show, deferring presentation for a short time, i.e. just a few seconds, can lead to a large mitigation of disruption.

Introduction

Interruption is becoming an increasingly common and frequent occurrence in human–computer interaction. E-mail notifications (Jackson, Dawson, & Wilson, 2001), instant messages (Cutrell, Czerwinski, & Horvitz, 2000), and agent-initiated interactions (Maes, 1994) are all contributing to a burgeoning epidemic of interruption at the user interface.

While application-initiated interruptions can be a nuisance, empirical investigation is needed to further quantify and better understand the effects of interruption (McFarlane & Latorella, 2002), to test models of cognitive processes that can reliably predict those effects (Gillie and Broadbent, 1989, McCrickard et al., 2003) and to motivate the need for and posit computational strategies that can mitigate disruption caused by interruption (Horvitz, Jacobs, & Hovel, 1999). The experiment reported in this paper contributes to these areas by further quantifying effects of interruption on both users and their tasks, comparing moments in a primary task sequence previously speculated to cause more or less disruption, and using the results to motivate the use of temporal strategies in attention-aware systems for mitigating the effects of interruption.

Our experiment utilized a commonly used paradigm where users perform primary tasks and are occasionally interrupted to perform peripheral tasks. It differs from previous interruption experiments because it measures effects along performance and affective dimensions, controls for task workload by manipulating only the time at which peripheral tasks are presented, and uses a theoretical basis for selecting the timings. Many prior experiments have measured the effects of interruption using some combination of completion time, error rate, and decision-making; including (Czerwinski et al., 2000a, Kreifeldt and McCarthy, 1981, Latorella, 1998, Monk et al., 2002, Speier et al., 1999, Trafton et al., 2003). In addition to completion time and error rate, our experiment measures effects of interruption along affective dimensions of annoyance and anxiety. Understanding these effects is particularly important for information workers and end users, as a few extra seconds or errors made on a task may be of low consequence relative to unnecessary increases in stress (Motowidlo, Packard, & Manning, 1986). While the interruption experiment conducted by Zijlstra, Roe, Leonora, and Krediet (1999) also measured affective dimensions, that experiment did not control for task workload. Our experiment controls for task workload by manipulating only the time at which peripheral tasks are presented to a user relative to a sequence of primary tasks.

We selected moments for interrupting a primary task sequence from theoretical arguments by Miyata and Norman (1986). The authors speculate that task boundaries represent more opportune moments for interruption because users have reduced mental workload at these moments. When a user completes a task, the executive system releases allocated resources, momentarily reducing workload before the cycle of allocation/deallocation occurs again for the next task. The boundary period from when resources are released to when resources are allocated for the next task should represent an opportune, or less disruptive, moment for interruption. Combining this hypothesis with a coarse structure of a primary task sequence, our experiment compares two moments for presenting peripheral tasks – between vs. during the execution of primary tasks.

For the experiment, we developed six categories of primary tasks: adding, counting, reading comprehension, image comprehension, selection, and registration, and two categories of peripheral tasks: reading news headlines and reasoning about market actions. Primary tasks were designed to be representative of tasks often performed within larger interactive activities and peripheral tasks were designed to be representative of information often maintained at the periphery of user attention. Users were divided into an Experimental and Control group. In the Experimental group, a user performed a sequence of three tasks from a primary category. The execution of one primary task was interrupted with a reading task, another was interrupted with a reasoning task, and the remaining task was not interrupted. This procedure was then repeated for the five remaining task categories. In the Control group, users performed the same primary tasks except that peripheral tasks were now presented at the boundaries between those tasks, which controlled for task workload. We measured completion time, error rate, annoyance, and anxiety, and validated difficulty of the primary tasks using subjective ratings.

Results show that when peripheral tasks interrupt the execution of primary tasks, users require more time to complete the primary tasks, commit more errors across tasks, and experience more annoyance and anxiety than when those same peripheral tasks are presented between the primary tasks. When extrapolated over the millions of computer users whose tasks are being increasingly interrupted by applications executing outside their focus of attention, our results show that the collective impact of these interruptions could be quite remarkable.

Since only the timing of peripheral tasks was manipulated, our results motivate the use of a temporal strategy in attention-aware systems for mitigating effects of interruption. By deferring delivery of peripheral information until coarse boundaries are reached during task execution, the resulting interruption would have considerably less disruptive impact. As shown by our results, a small delay in delivery could lead to a large mitigation of disruption.