Gaze-based predictive user interfaces: Visualizing user intentions in the presence of uncertainty
Introduction
For several years, great effort has been devoted to developing gaze-based prediction models that capture human behavior patterns naturally accompanying virtual interaction tasks such as reading an electronic document, or manipulating a virtual object (Fig. 1) (Bader, Vogelgesang, Klaus, 2009, Bednarik, Vrzakova, Hradis, 2012, Campbell, Maglio, 2001, Çığ, Sezgin, 2015, Courtemanche, Aïmeur, Dufresne, Najjar, Mpondo, 2011, Steichen, Carenini, Conati, 2013).
However, existing models are generally evaluated in terms of prediction accuracy, and within offline scenarios that assume perfect knowledge about user’s task-related intentions and goals. Such scenarios are called wizard-based test scenarios. Note that, in this paper, “online usage” does not refer to real-life usage scenarios. Online/offline distinction is made not based on how realistic the user interface is but based on whether the predictions are fed back to the user during interaction. In an example offline wizard-based test scenario, the users are asked to either select an object, or to manipulate a previously selected object (Bader et al., 2009). Collected data with labels corresponding to user intentions are then used to compute the accuracy of the related intention prediction model. The output of the prediction model is in no way shown to the users. In other words, in the wizard-based test scenarios, the loop between the user and the prediction system is open, i.e. the user is fed hardwired and perfect visual feedback via the user interface irrespective of predictions made by the prediction system (Fig. 2a). Existing studies do not take into account how these models would perform in the absence of wizards. They also do not examine how/if the prediction errors affect the quality of interaction. In this paper, we eliminate the wizard assumption and close the loop between the user and the prediction system. We achieve this by feeding highly accurate but imperfect predictions (since we do not have prediction systems that can perform with 100% accuracy yet) made by the prediction system to the user via appropriate visualizations of the user interface (Fig. 2b). By means of a thorough usability study, we seek answers to the following research questions: (1) How should a user interface adapt its behavior according to real-time predictions made by the underlying prediction system? (2) Will adaptations affect user behavior and inhibit performance of the prediction system (that assumes natural human behavior)? (3) Will prediction errors affect user behavior and inhibit performance of the prediction system? (4) Does users’ compatibility with the prediction system have implications for the design of such interfaces?
Section 2 gives a summary of related work on gaze-based predictive interfaces. Section 3 provides details on our usability study, proposed adaptive visualization approaches, and proposed gaze-based predictive user interfaces. Section 4 describes the evaluation of our predictive user interfaces in terms of performance, usability, and perceived task load. Section 5 concludes with a discussion of our work and a summary of future directions.
Section snippets
Related work
Explicit interfaces (e.g. text terminals and graphical user interfaces) rely on direct commands from the user to the computerized system. In contrast, implicit interfaces sense and reason about user actions that are not primarily aimed to interact with a computerized system to automatically trigger appropriate reactions (Schmidt, 2000). In order to reason about user actions with innovative sensors like eye trackers, implicit interfaces model human behavior by extracting useful and usable
Usability study
Consider the tasks described in Fig. 3. We have a gaze-based virtual task prediction system that can accurately distinguish between these tasks. In this paper, we propose to use online feedback from this system to build a user interface that dynamically adapts itself to user’s spontaneous task-related intentions and goals. This gives rise to the following research questions: (1) How should a user interface adapt its behavior according to real-time predictions made by the underlying prediction
Evaluation
We have proposed five different user interfaces. The first two are wizard-based interfaces. The first interface is the “gold standard” due to its deliberate resemblance to the WIMP-based user interfaces that users are accustomed to. More specifically, in this wizard interface, the underlying prediction system has no command over the interface and prediction results are not visualized by means of any interface adaptations. Expectedly, the user is unaware of predictions errors. Despite their
Future work and concluding remarks
We have presented the first line of work that uses online feedback from a gaze-based task prediction model to build a user interface that dynamically adapts itself to user’s spontaneous task-related intentions and goals. Since it is not yet possible to train prediction models that can perform with 100% accuracy, we have proposed novel approaches to providing visual feedback in the presence of uncertainty. From another point of view, we have closed the loop between the user and the prediction
Acknowledgments
The authors gratefully acknowledge the support and funding of TÜBİTAK (The Scientific and Technological Research Council of Turkey) under grant numbers 110E175 and 113E325 and TÜBA (Turkish Academy of Sciences).
References (25)
- et al.
Gaze-based prediction of pen-based virtual interaction tasks
Int. J. Hum.-Comput. Stud.
(2015) - et al.
Activity recognition using eye-gaze movements and traditional interactions
Interact. Comput.
(2011) - et al.
Gaze tutor: a gaze-reactive intelligent tutoring system
Int. J. Hum.-Comput. Stud.
(2012) - et al.
Development of nasa-tlx (task load index): results of empirical and theoretical research
Adv. Psychol.
(1988) - et al.
Multimodal integration of natural gaze behavior for intention recognition during object manipulation
Proceedings of the Eleventh International Conference on Multimodal Interfaces
(2009) - et al.
What do you want to do next: a novel approach for intent prediction in gaze-based interaction
Proceedings of the Symposium on Eye Tracking Research and Applications
(2012) Sus - a quick and dirty usability scale
Usability Eval. Ind.
(1996)- et al.
A robust algorithm for reading detection
Proceedings of the 2001 Workshop on Perceptive User Interfaces
(2001) - et al.
Highlighting interventions and user differences: informing adaptive information visualization support
Proceedings of the Thirty-second Annual ACM Conference on Human Factors in Computing Systems
(2014) - et al.
Real-time activity prediction: a gaze-based approach for early recognition of pen-based interaction tasks
Proceedings of the Twelfth Sketch-Based Interfaces and Modeling Symposium
(2015)
Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making
Comput. Graph. Forum
Cited by (18)
BIGaze: An eye-gaze action-guided Bayesian information gain framework for information exploration
2023, Advanced Engineering InformaticsWhat is supposed to help, actually hinders: about the impact of the GUI dynamics on the efficiency of gaze text entry
2023, Procedia Computer ScienceEffects of control-display gain and postural control method on distal pointing performance
2019, International Journal of Industrial ErgonomicsCitation Excerpt :Such a system would naturally give the user the feeling of being continuously and instantly successful (Bernardos et al., 2016; Jacob et al., 2008). Possible interaction methods for NUIs are speech (Gong, 1995), touch (Hofmeester and Wixon, 2010), gaze (Karaman and Sezgin, 2018), handwriting (Yanikoglu et al., 2017), or gesture-based interaction paradigms (LaViola, 1999; Oviatt et al., 2000). However, as Norman (2010) stated, the replacement of traditional interfaces with NUIs comes with “new problems, new challenges, and the potential for massive mistakes and confusion.”
A network analytic approach to gaze coordination during a collaborative task
2018, Computers in Human BehaviorCitation Excerpt :These results, and the use of ENA to model gaze behavior patterns more generally, suggest a number of potential learning applications. For example, these kinds of models could be used to better control the gaze behavior of artficial pedagogical agents—such as social robots and virtual characters—to improve gaze coordination in collaborative learning interactions between humans and agents (Admoni & Scassellati, 2017; Andrist, Gleicher, & Mutlu, 2017; Karaman & Sezgin, 2018). In particular, such models will aid in the development of anticipatory rather than reactive control mechanisms for agent behavior, improving coordination in collaborative contexts (Hayashi, 2016; Huang & Mutlu, 2016; Olsen et al., 2016).
A Human-Adaptive Model for User Performance and Fatigue Evaluation during Gaze-Tracking Tasks
2023, Electronics (Switzerland)Dwell Selection with ML-based Intent Prediction Using only Gaze Data
2022, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies