A fuzzy logics clustering approach to computing human attention allocation using eyegaze movement cue

https://doi.org/10.1016/j.ijhcs.2008.12.005Get rights and content

Abstract

Human's attention is an important element in human–machine interface design due to a close relationship between operator's attention and operator's work performance. However, understanding of operator's attention allocation while he or she is performing a task remains a challenging task because attention is generally unobservable, immeasurable, and uncertain. In our previous study, we demonstrated the effectiveness of using operator's eye movement information to understand attention allocation, which has made attention observable. The present paper describes our study which addressed immeasurability and uncertainty of attention. Specifically, we used eye fixation's duration to indicate operator's attention and developed a new computational model for the attention and its allocation using fuzzy logics clustering techniques. Along with the development of this model, we also developed an experiment to verify the effectiveness of the model. The result of the experiment shows that the model is promising.

Introduction

Despite its importance, understanding of operator's attention allocation behavior when he or she operates on an interface under a certain environment remains a challenge. The root causes of this challenge come from the inherent characteristics of human attention including (1) human attention is unobservable—there is no reliable way to “see” human attention (Harré, 2002; Gonalves et al., 2000), (2) human attention can hardly be measured or quantified—there is neither a physical unit to describe human attention nor a method to infer the state of attention (Wu et al., 2005; Horvitz et al., 2003), and (3) human attention is expressed in an uncertain or vague manner—there is no definite boundary between “concentration” and “distraction” for attention. In the case of attention to the interface through a visual modality, it is not possible to describe an exact region where attention falls (Brown et al., 2003).

With regard to the unobservability of human attention, the recent research suggests that the eye movement information can implicitly indicate the area of user's attention (Hess et al., 1998). There are several studies on using eye movement parameters to understand operator's attention allocation (Goldberg and Kotval, 1998; Smith et al., 2003; Lin et al., 2003; Chambers and Mattingley, 2005; Roda and Thomas, 2006; Le Meur et al., 2006). These studies have concluded (i) eye movements may yield important clues to human attention allocation at a fine temporal grain size typically on the order of 10 ms, (ii) data about eye movement can be collected non-intrusively, and (iii) eye movement parameters can serve as a sole source of information or as a supplement to other sources like verbal protocols (VP), electromyography (EMG), electroencephalogram (EEG), etc.

Measurement of human attention allocation based on eyegaze tracking has been studied by some other researchers (Treisman, 1986; Itti and Koch, 2001; Quek et al., 2002) including ours (Lin et al., 2003). Information from eye movements includes eye fixations, eye saccades, pupil size, blink rate, eye vergence, etc. Our previous study has shown that eye fixation is one of the important indicators among them (Lin et al., 2003). Eye fixations are pause in the eye scanning process over informative regions of interest. Measures of fixation include fixation duration—the time spent on an area of the visual field (also known as dwell time) and the number of fixations—the number of times the eye dwells on an area of the visual field. The longer fixation duration may imply the more time spent on cognition of a target. Eyegaze fixation is negatively correlated to the efficiency of task execution; particularly a more number of fixations imply that more information is required for performing a task. Eye fixation can thus be used to infer attention state as well as its allocation.

Attention allocation behavior is related to both task performance and interface design in such a way that given a particular interface design and a particular class of task, there will be a particular attention allocation behavior which further corresponds to task performance. As such, different interface designs can be evaluated in terms of attention allocation behavior. Furthermore, if a standard attention allocation behavior for a class of task is given, it is possible to adapt an interface to individuals to meet this standard behavior. This makes sense for adaptive interface to individuals for a class of task. Therefore, it is very useful to quantify attention allocation behavior, which is a primary motivation for our study.

Our study was based on an assertion that human's (visual) attention to a particular piece of knowledge or information displayed on an interface media will not be “crisp” or exclusive to that piece; in other words, the neighboring elements of that piece may occupy the attention as well. This assertion seems to be shared by Treisman (1986) who suggested that processing outside of focus is only “attenuated” (weakened) not stopped entirely. Thus, in our study, we considered that such a boundary is vague, and we developed a model to capture this vagueness by applying fuzzy logics clustering techniques. The model was applied to a plant operation problem and thus it was validated. Our model differs from those in the literature (Itti and Koch, 2001) in that we applied fuzzy logics clustering techniques which we believe to be the most natural fit.

The organization of the paper is as follows. Section 2 presents a model of attention based on eyegaze fixation. Section 3 presents a fuzzy logics model of human attention with respect to an entire interface. Section 4 presents a method to compute attention allocation strategy, and Section 5 presents a method to compute the pattern of attention which is further related to task performance. In Section 6, an experiment is presented to illustrate how our approach works and validate it. Section 7 concludes the paper with a brief discussion of future work.

Section snippets

A model of human attention with respect to visual elements

Our study assumed a particular task performing context where the attentional focus on a displayed element is exclusively in align with the element the eyegaze focuses. Fig. 1 shows a conceptual model of the visual attention based on this assumption where there is a single direct connection (which represents a perception from a perceptive of cognitive effectiveness) between the eye and interface (element)—called single eye–interface connection. Further, we considered that all points on the

A model for human attention with respect to an interface

Interface can be viewed as an artifact which is composed of a set of information elements or components which are connected in a certain way to bear semantics or meaning. When the eyegaze falls onto a component (which has meaning or semantics), we say that attention is established on this component—in other words, a cognitive activity is performed on the semantics of this component. Further, as we discussed before, it is inevitable that the eyegaze may be attracted by components surrounding

A model of attention allocation strategy

Human's cognition strategy is associated with attention allocation strategy. The attention allocation strategy is the attention distribution over interface components through a period of task performing processes or attention allocation processes. Since the attention to an interface at a particular time can be represented with Eq. (6), the attention allocation strategy in a period of task performing processes can then be represented by a matrix A=[AC1AC2ACl]. This matrix is similar to the

Computing attention allocation strategy pattern

Human's attention strategy is further related to human's performing a task and interface design, as the task is performed on the interface. Given an interface design, evaluation of the design requires that human subjects perform task on the interface. It is possible that we can find the patterns of operator's attention allocation strategy which are correlated to task performances. Such a relationship can help the human–machine interface design and human–machine interaction management in two

Human–machine interface: DURESS

A thermal-hydraulic process plant system called the Dual Reservoir System Simulation (DURESS) was taken as the example. DURESS was initially prototyped by Vicente (1991) for illustrating and validating the ecological interface design framework (Vicente and Rasmussen, 1992; Vicente et al., 1995); see Fig. 4. More detailed information of the system could be found in Lin et al. (2003) and Lin and Zhang (2004).

The DURESS system consists of the following components: VA and VB stand for input valve

Conclusion and future work

The study has made effort on developing a quantitative method to measure human visual attention allocation with consideration of vagueness in attention expression or representation. Further, the study has developed a method to extract human attention allocation pattern. An experiment was conducted to illustrate the proposed method and to provide validation for the proposed method. This study concludes (1) the proposed multiple eye–interface connection model is promising, (2) it is possible to

Acknowledgements

This research was made possible by the financial support from the National Sciences and Engineering Council of Canada (NSERC) and Atomic Energy Canada Limited through a CRD Grant (PI: Dr. W.J. Zhang). The continuing study is supported by NSERC through a Discovery Grant and University Faculty Award program awarded to Dr. Yingzi Lin. The authors—Drs. Lin and Zhang would like to specially acknowledge the following: (1) Dr. A.B. Thornton-Trump from the Biomechanics Lab, the University of Manitoba,

References (30)

  • E. Horvitz et al.

    Models of attention in computing and communication: from principles to applications

    Communications of ACM

    (2003)
  • Huang, X., Lin, Y., 2008. A vision-based hybrid method for facial expression recognition. In: International Conference...
  • Huffman, D.A., 1952. A method for the construction of minimum-redundancy codes. In: Proceedings of the I.R.E., pp....
  • L. Itti et al.

    Computational modeling of visual attention

    Nature Reviews Neuroscience

    (2001)
  • C.B. James

    Pattern Recognition with Fuzzy Objective Function Algorithms

    (1981)
  • Cited by (11)

    • An Adaptive Interface Design (AID) for enhanced computer accessibility and rehabilitation

      2017, International Journal of Human Computer Studies
      Citation Excerpt :

      When using wearable sensors, it is important to consider their degree of intrusiveness as they form an additional layer between the human-machine interactions (Lin, 2008). Non-contact sensors include vision-based sensors such as the tracking of body movements through computer vision (Micelli et al., 2009; Placidi, 2007; Betke et al., 2002; Pavlovic et al., 1995; https://www.leapmotion.com/; Basso Moro et al., 2014), visual attention tracking through eye-gaze monitoring (Lin et al., 2009, 2003; Breugelmans and Lin, 2011a, 2011b; Breugelmans et al., 2010, 2009) and video/camera systems for facial expressions (Pantic and Rothkrantz, 2003). For those applications that require contact-based sensing which is ubiquitous in human-machine interaction (Meng and Lee, 2006), Lin developed a new paradigm for biosensors called the natural-contact biosensor (Lin, 2011).

    • Augmented reality applications in design and manufacturing

      2012, CIRP Annals - Manufacturing Technology
      Citation Excerpt :

      In other areas such as HCI in learning, negative emotion has been shown to degrade a learner's cognitive performance [145]. Visual attention can be another important factor to be considered in designing the interfaces for AR tools [88]. Further in this stream, adaptive AR tools are promising.

    • Attention level determination by using fuzzy

      2020, International Journal of Integrated Engineering
    View all citing articles on Scopus
    View full text