Instantaneous Human-Computer Interactions: Button Causes and Screen Effects

Abstract

Many human-computer interactions are highly time-dependent, which means that an effect should follow a cause without delay. In this work, we explore how much time can pass between a cause and its effect without jeopardising the subjective perception of instantaneity. We ran two experiments that involve the same simple interaction: A click of a button causes a spinning disc to change its direction of rotation, following a variable delay. In our adjustment experiment, we asked participants to adjust the delay directly, but without numerical references, using repeated attempts to achieve a value as close to zero as possible. In the discrimination task, participants made judgements on whether the single rotation change happened immediately following the button-click, or after a delay. The derived thresholds revealed a marked difference between the two experimental approaches, participants could adjust delays down to a median of 40 ms, whereas the discrimination mid-point corresponded to 148 ms. This difference could possibly be an artefact of separate strategies adapted by participants for the two tasks. Alternatively, repeated presentations may make people more sensitive to delays, or provide them with additional information to base their judgements on. In either case, we have found that humans are capable of perceiving very short temporal delays, and these empirical results provide useful guidelines for future designs of time-critical interactions.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Several of our work and after-work hours are spent typing and clicking with keys and buttons to tell a machine what to show on a screen. Sometimes, we don’t pay much attention to the swiftness of the visual presentation; yet at times, we want the input to lead to instant results. However, due to screen refresh intervals, buffering and sometimes network stalls, our actions are not always immediately followed by the expected outcomes. The extent and acceptability of an outcome delay is highly context-dependent. In human-computer interactions, the responsiveness of a system can vary from a few milliseconds to a few seconds. Seow separates between four categories of responsiveness [1], labelling the slowest interactions flow and continuous; both of these exceed one second. Somewhat faster are immediate responses, which range from half a second to one second, and the fastest are termed instantaneous. The latter category is recommended for graphical controls and other interactions that mimic the physical world, but the thresholds are only estimated to be between 100 and 200 ms. While delays shorter than 100 ms may be imperceptible, they can still affect user performance, for instance through increased stress levels [2]. This work sets out to establish empirical values for how fast an outcome must follow an input in order for a user to perceive it as instantaneous.

1.1 Interaction Delays and Sensory Processes

Humans are very adept at handling and acting on objects, facilitated by both the motoric and the visual systems, along with other inputs. Sense of agency refers to the experience of being the direct cause of an event, and this term encompasses the expected delays that follow many actions [3]. Indeed, one study found that participants maintained the sense of agency from a joystick controlling the movements of an image for intervals as long as 700 ms [4]. This type of delay can approximate those found that follow real physical events, where consequences are stalled by the time taken to traverse a distance. However, many human-computer interactions involve series of inputs and outputs and these require far more speedy reactions. Whether typing in text, shooting at moving targets, or moving a cursor across the screen, most users expect instantaneous responses from the system. The higher demands for this type of human-computer interaction is emphasised by the findings of an experiment that compare the temporal boundaries for the sense of agency and the sense of simultaneity [5]. Participants were asked to push a button and watch for a visual flash, then make a judgement on the simultaneity of the events, or on the event serving as the agent. On average, the button-push was perceived as the agent as long as the visual flash did not lag by more than $\approx$400 ms; conversely, the two events were judged as simultaneous, at greater than chance rates, when the flash delay stayed below $\approx$250 ms [5]. In fast-paced game scenarios, similar delays become noticeable to players around 100 ms, and these can be detrimental to the gaming experience [6, 7]. Furthermore, mouse actions that require pointing and dragging have been found to be even more sensitive to temporal delays [8]. Still, humans are capable of adapting to fairly long temporal delays (235 ms) between movements of a mouse and movements on a screen, although this becomes increasingly difficult as the visual task speeds up [9].

Clearly, instantaneous and simultaneous are not synonymous with zero delay, a computational impossibility. Yet, these and similar human-computer interactions place strong demands for speedy responses on a system. Moreover, studies on multisensory and sensorimotor processes have demonstrated that the human perceptual system is adaptable and quite capable of compensating for short temporal offsets between corresponding signals [5, 10–12]. In our quest to find out exactly how much visual lag the perceptual system can compensate for following a motoric input, we have run a series of behavioural experiments on motor-visual delays. Our initial investigations involved direct delay adjustments using a jog-shuttle, with the corresponding visual event presented as disc that flashed on or off on a screen [13]. This approach allowed participants to repeatedly test and adjust the delay by turning the wheel and clicking the button of the jog-shuttle. Results from this experiment revealed that people vary greatly in their sensitivity to this type of delay, but the established median threshold was still far lower than expected at 39 ms. Adding system limitations to this value, our first investigation concluded that humans are on average capable of perceiving motor-visual delays as short as 51–90 ms.

1.2 Discrimination and Adjustment of Delays

This study continues our investigations into human sensitivity to motor-visual delays. It addresses the question on the appropriateness of our initial experimental approach and puts it back to back with a more traditional approach. Hence, we compare thresholds derived from two distinct methodologies, aiming for a more expansive range of data to generalise from. Because our first paradigm [13] allows participants some leeway to get results lower than they can actually perceive, we selected an isolated experimental task that relies on subjective discrimination and a binomial response selection. The simultaneity judgement task is a common methdology in multisensory research [5, 10, 12], and like the name implies, it involves a judgement call on the simultaneity of two signals. In our version of this task, participants are asked to discriminate between a motoric input and a visual output and make a judgement on whether the output followed immediately or whether it was delayed. Our comparison of delay thresholds established from two different methodologies thus forms the basis for an ongoing discussion around the use of less traditional experimental methods. Furthermore, we build on our earlier experiment where the motoric input, the button-click, resulted in the appearance or disappearance of a black disc [13]. Thus, we extend our work by adding dynamics to the previously static presentation. In the two current experiments, the visual presentation is made up of a black disc that rotates continuously at a steady pace. Moreover, bearing in mind that fast visual presentations can affect performance on these types of tasks [9], we include two speeds of rotation. By doing so, we explore whether the speed of motion can influence not only performance, but also the sensitivity to motor-visual delays.

2 Method

We explored subjective sensitivity to temporal delays between motor inputs and visual outputs in two repeated-measures experiments. The first experiment applied the described adjustment task, extending on our earlier work by replacing the static visual stimulus with a rotating disc. The second experiment encompassed the same visual presentation, but introduced a variation of the more common simultaneity judgement task [5, 10, 12], hereafter referred to as the discrimination task. We ran the two experiments over one session in a computer lab at Westerdals, with the order of presentation counterbalanced across our 10 female and 41 male participants (aged between 19 and 33 years).

Fig. 1.

Illustration of the experimental set-up.

We aimed to keep conditions as comparable as possible across the experiments, allowing for a direct comparison between the two methodologies. We therefore used the same visual stimulus throughout, simply a black disc moving in a continuous circle. The experiments ran on MacBook Pro computers with 15.4$^{\prime \prime }$ monitors and participants’ adjustments and responses were registered using Griffin click+spin USB controllers¹. These are simple controllers called jog-shuttles, comprised of a big click-button that also serves a rotating wheel. In both experiments, the button served as a trigger to change the direction of the disc’s rotation, as portrayed in Fig. 1. The disc rotated either slowly or quickly (0.2 or 1 revolution/s), with the speed of rotation varying randomly from trial to trial; the initial direction of rotation was also randomised across trials.

2.1 Adjustment Experiment

For the adjustment experiment, each trial commenced with an initial delay (100, 200, 300, or 400 ms). Participants were instructed to push the button on the jog-shuttle to change the direction of rotation and turn the wheel to adjust the delay between their push and the visual change. Due to the lack of reference points, participants were always unaware of the physical value of the delay; however, a clock-face served as a visual cue for the full range of delays. The task involved repeated adjustments and tests of the motor-visual delay, using the jog-shuttle, until the visual presentation was perceived to follow the motoric input instantaneously. Participants were allowed to spend as long as they wanted on each trial, but they had to make a minimum of ten button-clicks before proceeding to the next trial. Each trial therefore involved a series of wheel rotations and button-clicks before reaching the point of no delay, in a sequence illustrated in Fig. 2. With four levels of initial delay, two levels of rotation speed and two levels of rotation direction, along with two repetitions of all conditions, the full adjustment experiment included 32 trials and took approximately 10 min to complete.

Fig. 2.

Circular timeline illustrating the experimental procedure for the adjustment experiment. The experiment starts with an initial push of the button and a delayed change in rotation direction and it continues for as long as it takes the participant to adjust the delay down to an imperceptible level.

2.2 Discrimination Experiment

In the discrimination experiment, each trial involved a single button-click with a corresponding change in the direction of rotation. The delay between the click and the directional change varied randomly between 11 pre-established values (0, 20, 40, 60, 80, 100, 140, 180, 220, 260, and 300 ms). We asked participants to click button and pay close attention to the visual change. They were thereafter prompted for a judgement on the simultaneity of the motor and the visual events. Participants provided their responses by turning the wheel left or right to choose either the “immediate” or the “delayed” response options. We included four repetitions of all delay, direction and speed conditions, making a total of 176 experimental trials. With the short trial presentations, the experiment duration was on average 12 min.

2.3 Limitations

Our methodologies carry with them a few limitations. Even without system or network lags, computers will always introduce some delay in any interaction. These values are largely disregarded in the literature because they cannot be controlled in an experimental set-up. Although a computer’s internal system does not have the functionality to calculate the total duration between user input and screen output, we have applied an external set-up to measure this delay. Using a light sensor on the screen and an extra connector to the button, we performed 10 measurements and obtained an average input-output delay of 51 ms, this procedure is described in more detail in [14]. We report all scores and thresholds without adding this delay, in order to allow for comparisons with earlier studies in the field. However, this number should be kept in mind for a better representation of the human ability to detect motor-visual delays. Moreover, adding this delay to our results also improves the ground for comparison with findings from studies that make use of purpose-built experiment hardware.

3 Results

We initially treated data from the two experiments separately. For every factor and delay level, we calculated each participant’s mean and then derived the $50^\mathrm{th}$ percentile threshold from their individual distributions. This statistical approach did result in thresholds that exceeded the presented delay values, but only for a few individuals who likely have high tolerance to these types of delay. Furthermore, we took the precaution of checking for outliers based on the discrimination task distributions. For the vast majority of participants, the rate of “immediate” responses decreased as delays increased. However, one participant’s scores were discarded because the rate of “immediate” responses increased alongside the delay values, yielding a negative threshold value.

3.1 Adjustment Task

We ran two Wilcoxon signed rank-sum tests to investigate potential variations in delay sensitivity between the two initial disc rotation directions and the two disc rotation speeds, we also ran a Friedman test to explore differences due to the initial delay values. None of the tests revealed significant differences between the conditions. Following this, we collapsed scores across presentation modes and established the overall median threshold for adjusted delays to 40 ms. The $25^\mathrm{th}$, $50^\mathrm{th}$ and $75^\mathrm{th}$ percentile thresholds are presented as an empirical cumulative distribution in Fig. 3. With our motivation to evaluate the appropriateness of two distinct experimental methodologies, we also established the mode value for delay adjustments from the density plot presented in Fig. 5a. From this distribution, we found that the mode falls around 30 ms, a lower value than the median threshold. Furthermore, we observed an asymmetrical distribution, where the majority of scores centered around the mode, but a long tail of delay scores extended close to 400 ms.

Fig. 3.

Individual adjusted delay scores plotted as an empirical cumulative density distribution. The x-axis shows participants’ final accepted delay score and the y-axis corresponds to the proportion of scores that fall within defined range.

3.2 Discrimination Task

For the discrimination scores, we derived a best-fit logistic regression model, see Fig. 4. Running another two Wilcoxon signed rank-sum tests, we found no significant differences between the rotation directions and the rotation speeds. Hence, we collapsed scores across presentation modes and established the discrimination threshold from the mid-point between “immediate” and “delayed” responses, at 148 ms. As before, we established the mode value for the discrimination mid-points from their density plot, which is illustrated in Fig. 5b. Again the mode yielded a lower value than the median, this time it approximated 121 ms. The distribution showed a wide dispersion of scores around the mode, along with a long tail that ran to the end of the experimental range of 500 ms.

Fig. 4.

The best-fitting logistic regression line with the proportion of participants’ “immediate” responses plotted as a function of presented motor-visual delays.

3.3 Comparison of Experimental Methodologies

Running a Wilcoxon signed rank-sum test, we settled that the striking difference between the mid-point thresholds established from the two methodologies is statistically significant (W(49) = 1871, p $<$ 0.001). While the median for the adjusted delay scores came to 40 ms, the corresponding median for discrimination mid-points was more than 100 ms higher at 148 ms. The same difference was evident for the mode values, established at 30 ms and 121 ms, respectively. Furthermore, scores varied greatly across participants in both experiments. Yet, the wider dispersion of the density plot for discrimination mid-points suggests more individual variation for this task compared to the adjustment task.

4 Discussion

In this work, we have addressed the question on human sensitivity to motor-visual delays. This question has been considered by others before us, in the broad context of human-computer interactions [1], but also for fundemental multisensory processes [4, 5, 11] and for game-specific scenarios [6, 7]. We followed up this body of work with a study that explored isolated motor-visual interactions, focusing on the lower range of delay values. We set out to extend on our ongoing innvestigations by introducing dynamic visual stimuli. Furthermore, we sought to assess the generalisibility of results from our adjustment experiment.

In contrast to the higher ranges of motor-visual delays explored by others on this topic [1, 4, 9], we found that humans are capable of perceiving delays shorter than 100 ms. The thresholds derived from our adjustment experiment indicate that approximately half of the motor-visual adjustments yielded values equal to or below 40 ms, while a quarter of adjustment values fell around or below 15 ms. Keeping in mind that delays related to internal processes are often overlooked, we emphasise that these thresholds are under-estimates of the true delay. By adding the measured system delay to our results (outlined in Sect. 2.3), we include all known sources of delay. Thus we present our most representative thresholds, the $25^\mathrm{th}$, $50^\mathrm{th}$ and $75^\mathrm{th}$ percentiles at 66 ms, 91 ms and 136 ms.

Fig. 5.

Subjective delay thresholds presented as density plots

From our comparison of the two experimental procedures, we found that the discrimination experiment’s mid-point was more than 100 ms greater than the adjustment experiment’s median. With this marked difference, how do we deem which one is more representative of human delay sensitivity? The answer may lie in the context. A short delay can be difficult to perceive with just a single presentation, whereas repeated exposures provide several temporal reference points. Moreover, the discrimination task calls for a simple decision on the precense or absence of delay, whereas the adjustment task rests on the premise that there is a delay present and this should be adjusted down. Hence, the adjustment task dictates engagement from participants and it provides ample opportunity to move past points of uncertainty. On the other hand, the repetitive nature of the adjustment task could also allow participants to adapt a personal strategy to minimise the accepted delay. We observed examples of participants clicking the button quickly and steadily, making the disc bounce back and forth, so they could more easily judge the delay following the click. Relatedly, the repeated adjustments could give way for accepted delay values that fall below the subjective detection thresholds. If a participant manages to find two detection thresholds, one on either side of point zero, they could theoretically turn the wheel midway between the thresholds and accept a value very close to zero. Nevertheless, even when assuming that all pariticpants have adopted such a strategy, the extended range of delay values still fall below the discrimination threshold [13].

Furthermore, the adjustment experiment demonstrates internal validity from the constancy of the derived thresholds. We ran the first experiment with a static visual presentation and a 20 ms temporal resolution on the adjustment wheel. This time we introduced a moving visual presentation and we increased the temporal resolution for adjustments, before we ran the experiment on a new group of participants. Despite the changes, the $25^\mathrm{th}$, $50^\mathrm{th}$ and $75^\mathrm{th}$ percentiles are virtually identical across the two adjustment experiments. Although there is no ground for a similar comparison for the discrimination experiment, the density plots in Fig. 5 show a wider dispersion of subjective thresholds. Accordingly, the task of judging immediacy and delay had our participants accept higher delay values than they could adjust for, and they made their judgements with less consistency. As before, this outcome may be an artefact of strategies adopted by participants. Despite this note of warning, we find it remarkable that several of our participants are capable of manually tuning the motor-visual delays down to values below 100 ms, and do so repeatedly and congruously.

Many human-computer interactions involve more than a single, delayed output. Ongoing tasks that are carried out using a computer tend to involve series of inputs and outputs, similar to our adjustment task. Arguably, the adjustment experiment may be more representative of the scenarios we are interested in. However, what we perceive may not necessarily affect how we perform. In order to understand how motor-visual delays influence not only the conscious experience, but also the interaction itself, we need to evaluate the ability to compensate for delays when performing a task. Thus, we plan to apply the derived range of subjective thresholds, which covers a fairly large sample’s sensitiviy to motor-visual delays, to the study of performance on motor-visual tasks. In so doing, we aim to shed more light on which experimental approach provides the most representative estimate of motor-visual temporal sensitivity, and we hope to find out whether perceptible and imperceptible delays can affect performance.

5 Concluding Remarks and Future Work

This study presents findings on human sensitivity to delays between a button-click and a visual presentation, which show that repeated motor-visual interactions can make inherent delays more noticeable. Our results also demonstrate that the dynamics of a visual presentation has little impact on the perceived delay that precedes it. Instead, variations are far greater between individuals. The most sensitive of our participants contributed to establish the $25^\mathrm{th}$ percentile at 15 ms, or 66 ms when adding internal system delays. Considering that the median also falls below 100 ms, these outcomes speak in favour of designing interactive systems with very fast responses. Ideally, no user should be able to notice delays during interactions with a computer and the presented thresholds highlight the challenge of meeting these demands.

So far, our investigations have focused on universal thresholds for motor-visual delay sensitivity. Our work shows that this sensitivity varies greatly between individuals, and we wish to explore potential factors that could influence the subjective perception of temporal delays. In particular, we plan to look into earlier encounters with highly time-dependent processes, such as gaming and musical experience. Additionally, some of the work on motor-visual delay addresses the relevant scenarios directly; for instance, the work of Claypool and colleagues focus on delay in real games [6]. Conversely, we commenced our investigations with simple and isolated motor-visual interactions. In this work, we have extended on our earlier study and added dynamics to the previously static visual presentation. This had very little influence on the derived thresholds for the delay adjustment task. The final step in our on-going work will be to apply motor-visual delays to an interactive task, a simple game, to explore whether task performance is affected by barely noticeable delays.