Pupil responses to continuous aiming movements

Highlights

• Pupil responses correlate to task difficulty in continuous aiming movements.

• Pupil size changes can indicate the mental workload evoked by physical tasks.

• Movement-related pupil response was detected from self-paced visual−motor tasks.

• The task difficulty of the motor task was quantitatively defined using Fitts’ law.

• The pupillary measure was expanded from cognitive workload to physical workload.

Abstract

Pupillary response is associated with perceptual and cognitive loads in visual and cognitive tasks, but no quantitative link between pupil response and the task workload in visual−motor tasks has been confirmed. The objective of this study is to investigate how the changes of task requirement of a visual−motor task are reflected by the changes of pupil size. In the present study, a simple continuous aiming task is performed and the task requirement is manipulated and measured by Fitts’ Index of Difficulty (ID), calculated for different combinations of the target size and movement distance. Pupil response is recorded using a remote eye-tracker. The results show that event-triggered pupil dilations in continuous aiming movements respect Fitts’ Law, such that higher task difficulty evokes higher peak pupil dilation and longer peak duration. These findings suggest that pupil diameter can be employed as a physiological indicator to task workload evoked by the task requirement in visual−motor tasks.

Introduction

Evaluating the task workload of visual−motor tasks, and specifically the tasks’ impact on the mental load of the user, is of great importance in monitoring and managing the workload in various tasks and systems, such as designing human−machine interfaces (Bailey and Iqbal, 2008, Goldberg and Kotval, 1999, Iqbal et al., 2005, Pomplun and Sunkara, 2003) and evaluating the mental workload of operators in high skill demanding work environments such as surgeons (Carswell et al., 2005, Zheng et al., 2010) and vehicle and aviation operators (Kun et al., 2012, Palinko and Kun, 2012, Veltman and Gaillard, 1998). Pupillary response has been extensively investigated and found to be a reliable indicator for the changes of cognitive loads in various cognitive tasks such as mental arithmetic and memory recall (Ahern and Jackson, 1979, Hess and Polt, 1964, Kahneman and Jackson, 1966); however, the relationship between pupil response and the changes of mental workload induced by physical demands in visual−motor tasks such as goal-directed movements have not been thoroughly explored. The confirmation of such relationship would expand the ability of using pupil diameter to indicate mental workload in visual−motor tasks.

The pioneering work, conducted by Richer and his colleagues in the 1980s (Richer and Beatty, 1985, Richer et al., 1983), examined the pupil response to a simple finger flexion movement during key-pressing. They found a connection between pupillary response and the task complexity; when the subjects were required to press buttons with increasing number of fingers, the amplitude of pupil response increased. Typically, the pupil started to dilate around 1.5 s before the finger movement and the pupil reached its peak size 0.5–1.0 s after the movement. However, Richer׳s task, only involving various numbers of fingers in the movement, was not a testing of real-world eye−hand coordination; the participants’ eyes fixed at the center of the screen throughout the task for the purpose of pupil size recording and isolating perceptual load out of movement. In ordinary everyday motor tasks, eyes are usually involved and the movements are continuous and complex, such as walking, driving, and playing ball games (Land, 2006). Furthermore, the task difficulty of Richer׳s study did not have a quantitative definition of the task requirement, i.e. the task difficulty was represented by the number of the fingers involved in the flexion instead of being scaled such as using Fitts’ Index of Difficulty.

To investigate the quantitative association between pupillary responses and the task requirement of motor tasks, we adopted the concept from Fitts’ study of the information processing model between environmental stimulation and human response. In aiming tasks, Fitts defined task requirement using the Index of Difficulty (ID), where increasing ID is predicted by increases of tool travel distance and decreases in target size, and the performance (movement time) is correlated with the ID, which is named Fitts’ law (Fitts, 1954, Fitts and Peterson, 1964). Fitts’ law is a fundamental method of quantitating task difficulty evaluation in HCI research and design due to its strong predictive power (Kopper et al., 2010, Mackenzie, 1992).

Before examining the pupil responses to the task requirement in continuous pointing movements, we examined the pupil responses to the task requirement in a discrete Fitts’ pointing task (Jiang et al., 2014a), where the subjects were required to move a tool to touch a circle with varying target sizes and distances and every movement was preceded by a 10 s wait. We found a small but significant dilation starting about 1.5 s before the tool started to move, followed by a slight constriction, the “valley” in the pupil size profile. Before the tool touched the target (2 s after the tool starts to leave), the pupil reached its peak size. Both the pupil dilation and the duration from Valley-to-Peak size positively correlate with the increase of IDs. This evidence indicates that the change of pupil diameter is regulated by task requirement. This finding was confirmed by a second study to determine whether the target size or target distance has an independent influence on the pupil response (Jiang et al., 2014b).

The above two studies documented the connection between pupillary response and task difficulty in discrete pointing tasks. The subjects were instructed to wait 10 s before taking the next aiming movement to ensure that the recorded pupil response would not be affected by the previous movement, and the pupil had time to return its baseline size. Examples of discrete visual−motor tasks in daily life include inserting a key into a lock, shooting a basketball, and mouse-clicking at a specific location in an editor. However, in reality of everyday interactive tasks, continuous tasks are more common, such as steering a vehicle, playing ping-pong, and selecting an item in a multiple-level cascade menu. In many cases, the continuous movement frequency is higher than the pupil response frequency which is typically lower than 0.5 Hz (Jiang et al., 2014a, Richer and Beatty, 1985), pupil response is inevitably affected by multiple movements. It is a high time to carefully examine pupil response and develop a method to distinguish if pupil response is a reaction to an upcoming movement or is just a residual effect from a previous movement. We therefore explore the pupil responses to the change of task requirements in a continuous movement such as continuous aiming tasks, with the following research questions in mind. First, is there a difference between the patterns of pupil size responses between discrete and continuous visual−motor tasks? Second, is the change of pupil size still able to distinguish task difficulty in continuous visual−motor tasks?

We conducted the present study using a similar experimental setting as that in the discrete movement study (Jiang et al., 2014a) but here the participants performed a continuous pointing task without an extra waiting time between movements. We hypothesized that the pupil dilation will respect Fitts’ Law in continuous movements, such that higher task difficulty evokes higher peak pupil dilation and longer peak duration. If the hypothesis holds, it may be possible to employ pupil diameter as a reliable physiological indicator to quantitatively measure task workload in continuous visually-guided motor tasks. Such measurements can be used for continuously adjusting proactive responses of user interfaces, for example in medical educational simulations involving visual−motor tasks.