A fuzzy logics clustering approach to computing human attention allocation using eyegaze movement cue
Introduction
Despite its importance, understanding of operator's attention allocation behavior when he or she operates on an interface under a certain environment remains a challenge. The root causes of this challenge come from the inherent characteristics of human attention including (1) human attention is unobservable—there is no reliable way to “see” human attention (Harré, 2002; Gonalves et al., 2000), (2) human attention can hardly be measured or quantified—there is neither a physical unit to describe human attention nor a method to infer the state of attention (Wu et al., 2005; Horvitz et al., 2003), and (3) human attention is expressed in an uncertain or vague manner—there is no definite boundary between “concentration” and “distraction” for attention. In the case of attention to the interface through a visual modality, it is not possible to describe an exact region where attention falls (Brown et al., 2003).
With regard to the unobservability of human attention, the recent research suggests that the eye movement information can implicitly indicate the area of user's attention (Hess et al., 1998). There are several studies on using eye movement parameters to understand operator's attention allocation (Goldberg and Kotval, 1998; Smith et al., 2003; Lin et al., 2003; Chambers and Mattingley, 2005; Roda and Thomas, 2006; Le Meur et al., 2006). These studies have concluded (i) eye movements may yield important clues to human attention allocation at a fine temporal grain size typically on the order of 10 ms, (ii) data about eye movement can be collected non-intrusively, and (iii) eye movement parameters can serve as a sole source of information or as a supplement to other sources like verbal protocols (VP), electromyography (EMG), electroencephalogram (EEG), etc.
Measurement of human attention allocation based on eyegaze tracking has been studied by some other researchers (Treisman, 1986; Itti and Koch, 2001; Quek et al., 2002) including ours (Lin et al., 2003). Information from eye movements includes eye fixations, eye saccades, pupil size, blink rate, eye vergence, etc. Our previous study has shown that eye fixation is one of the important indicators among them (Lin et al., 2003). Eye fixations are pause in the eye scanning process over informative regions of interest. Measures of fixation include fixation duration—the time spent on an area of the visual field (also known as dwell time) and the number of fixations—the number of times the eye dwells on an area of the visual field. The longer fixation duration may imply the more time spent on cognition of a target. Eyegaze fixation is negatively correlated to the efficiency of task execution; particularly a more number of fixations imply that more information is required for performing a task. Eye fixation can thus be used to infer attention state as well as its allocation.
Attention allocation behavior is related to both task performance and interface design in such a way that given a particular interface design and a particular class of task, there will be a particular attention allocation behavior which further corresponds to task performance. As such, different interface designs can be evaluated in terms of attention allocation behavior. Furthermore, if a standard attention allocation behavior for a class of task is given, it is possible to adapt an interface to individuals to meet this standard behavior. This makes sense for adaptive interface to individuals for a class of task. Therefore, it is very useful to quantify attention allocation behavior, which is a primary motivation for our study.
Our study was based on an assertion that human's (visual) attention to a particular piece of knowledge or information displayed on an interface media will not be “crisp” or exclusive to that piece; in other words, the neighboring elements of that piece may occupy the attention as well. This assertion seems to be shared by Treisman (1986) who suggested that processing outside of focus is only “attenuated” (weakened) not stopped entirely. Thus, in our study, we considered that such a boundary is vague, and we developed a model to capture this vagueness by applying fuzzy logics clustering techniques. The model was applied to a plant operation problem and thus it was validated. Our model differs from those in the literature (Itti and Koch, 2001) in that we applied fuzzy logics clustering techniques which we believe to be the most natural fit.
The organization of the paper is as follows. Section 2 presents a model of attention based on eyegaze fixation. Section 3 presents a fuzzy logics model of human attention with respect to an entire interface. Section 4 presents a method to compute attention allocation strategy, and Section 5 presents a method to compute the pattern of attention which is further related to task performance. In Section 6, an experiment is presented to illustrate how our approach works and validate it. Section 7 concludes the paper with a brief discussion of future work.
Section snippets
A model of human attention with respect to visual elements
Our study assumed a particular task performing context where the attentional focus on a displayed element is exclusively in align with the element the eyegaze focuses. Fig. 1 shows a conceptual model of the visual attention based on this assumption where there is a single direct connection (which represents a perception from a perceptive of cognitive effectiveness) between the eye and interface (element)—called single eye–interface connection. Further, we considered that all points on the
A model for human attention with respect to an interface
Interface can be viewed as an artifact which is composed of a set of information elements or components which are connected in a certain way to bear semantics or meaning. When the eyegaze falls onto a component (which has meaning or semantics), we say that attention is established on this component—in other words, a cognitive activity is performed on the semantics of this component. Further, as we discussed before, it is inevitable that the eyegaze may be attracted by components surrounding
A model of attention allocation strategy
Human's cognition strategy is associated with attention allocation strategy. The attention allocation strategy is the attention distribution over interface components through a period of task performing processes or attention allocation processes. Since the attention to an interface at a particular time can be represented with Eq. (6), the attention allocation strategy in a period of task performing processes can then be represented by a matrix . This matrix is similar to the
Computing attention allocation strategy pattern
Human's attention strategy is further related to human's performing a task and interface design, as the task is performed on the interface. Given an interface design, evaluation of the design requires that human subjects perform task on the interface. It is possible that we can find the patterns of operator's attention allocation strategy which are correlated to task performances. Such a relationship can help the human–machine interface design and human–machine interaction management in two
Human–machine interface: DURESS
A thermal-hydraulic process plant system called the Dual Reservoir System Simulation (DURESS) was taken as the example. DURESS was initially prototyped by Vicente (1991) for illustrating and validating the ecological interface design framework (Vicente and Rasmussen, 1992; Vicente et al., 1995); see Fig. 4. More detailed information of the system could be found in Lin et al. (2003) and Lin and Zhang (2004).
The DURESS system consists of the following components: VA and VB stand for input valve
Conclusion and future work
The study has made effort on developing a quantitative method to measure human visual attention allocation with consideration of vagueness in attention expression or representation. Further, the study has developed a method to extract human attention allocation pattern. An experiment was conducted to illustrate the proposed method and to provide validation for the proposed method. This study concludes (1) the proposed multiple eye–interface connection model is promising, (2) it is possible to
Acknowledgements
This research was made possible by the financial support from the National Sciences and Engineering Council of Canada (NSERC) and Atomic Energy Canada Limited through a CRD Grant (PI: Dr. W.J. Zhang). The continuing study is supported by NSERC through a Discovery Grant and University Faculty Award program awarded to Dr. Yingzi Lin. The authors—Drs. Lin and Zhang would like to specially acknowledge the following: (1) Dr. A.B. Thornton-Trump from the Biomechanics Lab, the University of Manitoba,
References (30)
- et al.
Neurodisruption of selective attention: insights and implications
Trends in Cognitive Sciences
(2005) - et al.
Using eye movement parameters for evaluating human–machine interface frameworks under normal control operation and fault detection situation
International Journal of Human–Computer Studies
(2003) - et al.
Towards a novel interface design framework: function–behavior–state paradigm
International Journal of Human–Computer Studies
(2004) - et al.
Visual importance-biased image synthesis animation
- et al.
Pattern Classification
(2001) - et al.
Eye movement-based evaluation of the computer interface
- et al.
Tracing patterns and attention: humanoid robot cognition
IEEE Intelligent Systems Magazine
(2000) Cognitive Science
(2002)- et al.
Driving system adaptation: disambiguating non-invasive physiological measures with cognitive task analysis
- et al.
Fuzzy Cluster Analysis
(1999)
Models of attention in computing and communication: from principles to applications
Communications of ACM
Computational modeling of visual attention
Nature Reviews Neuroscience
Pattern Recognition with Fuzzy Objective Function Algorithms
Cited by (11)
MFCSNet: A Musician–Follower Complex Social Network for Measuring Musical Influence
2024, Entertainment ComputingAn Adaptive Interface Design (AID) for enhanced computer accessibility and rehabilitation
2017, International Journal of Human Computer StudiesCitation Excerpt :When using wearable sensors, it is important to consider their degree of intrusiveness as they form an additional layer between the human-machine interactions (Lin, 2008). Non-contact sensors include vision-based sensors such as the tracking of body movements through computer vision (Micelli et al., 2009; Placidi, 2007; Betke et al., 2002; Pavlovic et al., 1995; https://www.leapmotion.com/; Basso Moro et al., 2014), visual attention tracking through eye-gaze monitoring (Lin et al., 2009, 2003; Breugelmans and Lin, 2011a, 2011b; Breugelmans et al., 2010, 2009) and video/camera systems for facial expressions (Pantic and Rothkrantz, 2003). For those applications that require contact-based sensing which is ubiquitous in human-machine interaction (Meng and Lee, 2006), Lin developed a new paradigm for biosensors called the natural-contact biosensor (Lin, 2011).
Augmented reality applications in design and manufacturing
2012, CIRP Annals - Manufacturing TechnologyCitation Excerpt :In other areas such as HCI in learning, negative emotion has been shown to degrade a learner's cognitive performance [145]. Visual attention can be another important factor to be considered in designing the interfaces for AR tools [88]. Further in this stream, adaptive AR tools are promising.
Model study of target discrimination in concurrent auditory events
2022, Cognitive Computation and SystemsAttention level determination by using fuzzy
2020, International Journal of Integrated Engineering