Gaze-based predictive user interfaces: Visualizing user intentions in the presence of uncertainty

https://doi.org/10.1016/j.ijhcs.2017.11.005Get rights and content

Highlights

  • We propose two novel gaze-based predictive user interfaces.

  • Our interfaces are able to dynamically provide adaptive interventions.

  • Interventions reflect user’s task-related intentions and goals.

  • The presence of uncertainty in prediction model outputs is handled.

  • Usability and perceived task load are not adversely affected.

Abstract

Human eyes exhibit different characteristic patterns during different virtual interaction tasks such as moving a window, scrolling a piece of text, or maximizing an image. Human-computer studies literature contains examples of intelligent systems that can predict user’s task-related intentions and goals based on eye gaze behavior. However, these systems are generally evaluated in terms of prediction accuracy, and on previously collected offline interaction data. Little attention has been paid to creating real-time interactive systems using eye gaze and evaluating them in online use. We have five main contributions that address this gap from a variety of aspects. First, we present the first line of work that uses real-time feedback generated by a gaze-based probabilistic task prediction model to build an adaptive real-time visualization system. Our system is able to dynamically provide adaptive interventions that are informed by real-time user behavior data. Second, we propose two novel adaptive visualization approaches that take into account the presence of uncertainty in the outputs of prediction models. Third, we offer a personalization method to suggest which approach will be more suitable for each user in terms of system performance (measured in terms of prediction accuracy). Personalization boosts system performance and provides users with the more optimal visualization approach (measured in terms of usability and perceived task load). Fourth, by means of a thorough usability study, we quantify the effects of the proposed visualization approaches and prediction errors on natural user behavior and the performance of the underlying prediction systems. Finally, this paper also demonstrates that our previously-published gaze-based task prediction system, which was assessed as successful in an offline test scenario, can also be successfully utilized in realistic online usage scenarios.

Introduction

For several years, great effort has been devoted to developing gaze-based prediction models that capture human behavior patterns naturally accompanying virtual interaction tasks such as reading an electronic document, or manipulating a virtual object (Fig. 1) (Bader, Vogelgesang, Klaus, 2009, Bednarik, Vrzakova, Hradis, 2012, Campbell, Maglio, 2001, Çığ, Sezgin, 2015, Courtemanche, Aïmeur, Dufresne, Najjar, Mpondo, 2011, Steichen, Carenini, Conati, 2013).

However, existing models are generally evaluated in terms of prediction accuracy, and within offline scenarios that assume perfect knowledge about user’s task-related intentions and goals. Such scenarios are called wizard-based test scenarios. Note that, in this paper, “online usage” does not refer to real-life usage scenarios. Online/offline distinction is made not based on how realistic the user interface is but based on whether the predictions are fed back to the user during interaction. In an example offline wizard-based test scenario, the users are asked to either select an object, or to manipulate a previously selected object (Bader et al., 2009). Collected data with labels corresponding to user intentions are then used to compute the accuracy of the related intention prediction model. The output of the prediction model is in no way shown to the users. In other words, in the wizard-based test scenarios, the loop between the user and the prediction system is open, i.e. the user is fed hardwired and perfect visual feedback via the user interface irrespective of predictions made by the prediction system (Fig. 2a). Existing studies do not take into account how these models would perform in the absence of wizards. They also do not examine how/if the prediction errors affect the quality of interaction. In this paper, we eliminate the wizard assumption and close the loop between the user and the prediction system. We achieve this by feeding highly accurate but imperfect predictions (since we do not have prediction systems that can perform with 100% accuracy yet) made by the prediction system to the user via appropriate visualizations of the user interface (Fig. 2b). By means of a thorough usability study, we seek answers to the following research questions: (1) How should a user interface adapt its behavior according to real-time predictions made by the underlying prediction system? (2) Will adaptations affect user behavior and inhibit performance of the prediction system (that assumes natural human behavior)? (3) Will prediction errors affect user behavior and inhibit performance of the prediction system? (4) Does users’ compatibility with the prediction system have implications for the design of such interfaces?

Section 2 gives a summary of related work on gaze-based predictive interfaces. Section 3 provides details on our usability study, proposed adaptive visualization approaches, and proposed gaze-based predictive user interfaces. Section 4 describes the evaluation of our predictive user interfaces in terms of performance, usability, and perceived task load. Section 5 concludes with a discussion of our work and a summary of future directions.

Section snippets

Related work

Explicit interfaces (e.g. text terminals and graphical user interfaces) rely on direct commands from the user to the computerized system. In contrast, implicit interfaces sense and reason about user actions that are not primarily aimed to interact with a computerized system to automatically trigger appropriate reactions (Schmidt, 2000). In order to reason about user actions with innovative sensors like eye trackers, implicit interfaces model human behavior by extracting useful and usable

Usability study

Consider the tasks described in Fig. 3. We have a gaze-based virtual task prediction system that can accurately distinguish between these tasks. In this paper, we propose to use online feedback from this system to build a user interface that dynamically adapts itself to user’s spontaneous task-related intentions and goals. This gives rise to the following research questions: (1) How should a user interface adapt its behavior according to real-time predictions made by the underlying prediction

Evaluation

We have proposed five different user interfaces. The first two are wizard-based interfaces. The first interface is the “gold standard” due to its deliberate resemblance to the WIMP-based user interfaces that users are accustomed to. More specifically, in this wizard interface, the underlying prediction system has no command over the interface and prediction results are not visualized by means of any interface adaptations. Expectedly, the user is unaware of predictions errors. Despite their

Future work and concluding remarks

We have presented the first line of work that uses online feedback from a gaze-based task prediction model to build a user interface that dynamically adapts itself to user’s spontaneous task-related intentions and goals. Since it is not yet possible to train prediction models that can perform with 100% accuracy, we have proposed novel approaches to providing visual feedback in the presence of uncertainty. From another point of view, we have closed the loop between the user and the prediction

Acknowledgments

The authors gratefully acknowledge the support and funding of TÜBİTAK (The Scientific and Technological Research Council of Turkey) under grant numbers 110E175 and 113E325 and TÜBA (Turkish Academy of Sciences).

References (25)

  • Ç. Çığ et al.

    Gaze-based prediction of pen-based virtual interaction tasks

    Int. J. Hum.-Comput. Stud.

    (2015)
  • F. Courtemanche et al.

    Activity recognition using eye-gaze movements and traditional interactions

    Interact. Comput.

    (2011)
  • S. D’Mello et al.

    Gaze tutor: a gaze-reactive intelligent tutoring system

    Int. J. Hum.-Comput. Stud.

    (2012)
  • S.G. Hart et al.

    Development of nasa-tlx (task load index): results of empirical and theoretical research

    Adv. Psychol.

    (1988)
  • T. Bader et al.

    Multimodal integration of natural gaze behavior for intention recognition during object manipulation

    Proceedings of the Eleventh International Conference on Multimodal Interfaces

    (2009)
  • R. Bednarik et al.

    What do you want to do next: a novel approach for intent prediction in gaze-based interaction

    Proceedings of the Symposium on Eye Tracking Research and Applications

    (2012)
  • J. Brooke

    Sus - a quick and dirty usability scale

    Usability Eval. Ind.

    (1996)
  • C.S. Campbell et al.

    A robust algorithm for reading detection

    Proceedings of the 2001 Workshop on Perceptive User Interfaces

    (2001)
  • G. Carenini et al.

    Highlighting interventions and user differences: informing adaptive information visualization support

    Proceedings of the Thirty-second Annual ACM Conference on Human Factors in Computing Systems

    (2014)
  • Ç. Çığ et al.

    Real-time activity prediction: a gaze-based approach for early recognition of pen-based interaction tasks

    Proceedings of the Twelfth Sketch-Based Interfaces and Modeling Symposium

    (2015)
  • C. Conati et al.

    Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making

    Comput. Graph. Forum

    (2014)
  • DeBarr, D., 2006. Constrained dynamic time warping distance measure....
  • Cited by (18)

    • Effects of control-display gain and postural control method on distal pointing performance

      2019, International Journal of Industrial Ergonomics
      Citation Excerpt :

      Such a system would naturally give the user the feeling of being continuously and instantly successful (Bernardos et al., 2016; Jacob et al., 2008). Possible interaction methods for NUIs are speech (Gong, 1995), touch (Hofmeester and Wixon, 2010), gaze (Karaman and Sezgin, 2018), handwriting (Yanikoglu et al., 2017), or gesture-based interaction paradigms (LaViola, 1999; Oviatt et al., 2000). However, as Norman (2010) stated, the replacement of traditional interfaces with NUIs comes with “new problems, new challenges, and the potential for massive mistakes and confusion.”

    • A network analytic approach to gaze coordination during a collaborative task

      2018, Computers in Human Behavior
      Citation Excerpt :

      These results, and the use of ENA to model gaze behavior patterns more generally, suggest a number of potential learning applications. For example, these kinds of models could be used to better control the gaze behavior of artficial pedagogical agents—such as social robots and virtual characters—to improve gaze coordination in collaborative learning interactions between humans and agents (Admoni & Scassellati, 2017; Andrist, Gleicher, & Mutlu, 2017; Karaman & Sezgin, 2018). In particular, such models will aid in the development of anticipatory rather than reactive control mechanisms for agent behavior, improving coordination in collaborative contexts (Hayashi, 2016; Huang & Mutlu, 2016; Olsen et al., 2016).

    • Dwell Selection with ML-based Intent Prediction Using only Gaze Data

      2022, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    View all citing articles on Scopus
    View full text