The review of the literature on gesture design, the identification of the factors that are important to consider when designing gesture vocabularies, and the analysis of how different methodologies do or do not support the evaluation of different factors, suggested that there are many research avenues that should be considered to improve the design of gestures. In what follows, we present a potential gesture design process that could be used to iteratively evaluate a subset of factors that a designer deems appropriate for a given use case. We also discuss some additional factors that arose during the literature review and highlight how they could impact the ever-changing ecosystem of gesture design.
5.1 A Potential User-Centered, Factor-Centric Gesture Design Process
As indicated above, one limitation of existing gesture design approaches is that there is a mismatch between the sets of factors a methodology is suitable for and the sets of factors gesture designers may wish to optimize. Rather than selecting one methodology and being unable to evaluate a given factor, one possible alternative is to take a user-centered, factor-centric approach, wherein one uses a process that enables different factors to be iteratively refined, evaluated, or removed during design time, whilst respecting the inter-dependencies that exist between various factors.
One possible instantiation of such a design process is depicted in Figure
2. Within this example process, there are four phases that a designer would work through while designing their gesture vocabulary: identifying the preconditions that will dictate the user experience with the gestures, assigning prioritization weights to each of the evaluation factors based on the requirements from the preconditions, creating an initial gesture design using an existing methodology (or using a stock gesture vocabulary), continually refining and evaluating the vocabulary keeping in mind the weights that were assigned to different factors, and finally, arriving at the final gesture vocabulary.
5.1.1 Understanding Requirements and Prioritizing Factors.
With such an approach (Figure
2), designers would first determine and understand the requirements of their gesture vocabulary. In this example, user experience factors such as context, modality, recognition, and feedback, which will influence the user experience and the designer’s goals, are examined to determine the degree to which other factors should be weighed when iterating on a gesture vocabulary. For example, if a system is to be used with a mall kiosk, the discoverability of the gestures might be weighed higher than ergonomics because a given gesture may be performed infrequently. The identification of requirements will not only help designers form a complete understanding of the goals that are guiding their user experience, but also serve as a sanity check to ensure that a designer is aware of all the challenges their gesture vocabulary may encounter.
As many have demonstrated, there are tradeoffs between many of the factors that were identified in [
224,
266,
330,
335]. As these tradeoffs make it impossible to design a gesture vocabulary that would fulfill all the requirements of the factors, a designer should then prioritize the evaluation factors (and reconsider the effectiveness of the feedback and the recognition accuracy), given the requirements of the preconditions. One way to do this would be for a designer to assign each factor an importance descriptor, such as
high,
medium, or
low [
353]. As an example, if the end user population is blind, then it may be appropriate to rate factors such as Occlusion or Social Acceptability low, whereas factors such as Discoverability and Recognition should be rated high.
It is also worth noting that executing the various evaluation and refinement methodologies also requires time and resources (e.g., equipment and algorithms). Therefore, we recommend gesture designers consider the accessibility and suitability of the different methodologies regarding each of the factors when adopting this new gesture design approach.
5.1.2 Iteratively Evaluating and Refining Gesture Vocabulary.
After prioritizing the factors, a designer could then use their preferred method of choice to seed an initial gesture vocabulary (e.g., using a User-led, Expert-led, Computationally-based methodology, or using an existing, stock gesture vocabulary). Once the initial gesture vocabulary has been developed, the designer could then work through each of the factors based on their priority, iteratively evaluating, refining, and modifying the proposed gesture vocabulary using the methods summarized in Section
3 and Table
2. When a tradeoff must occur, factors with high or medium ratings should receive priority over lower rated ones. Factors that are interrelated, such as Complexity and Efficiency, should be evaluated holistically so that any changes that result do not cause a cascading effect in subsequent iterations over the gesture vocabulary.
The point at which each factor is considered should also be determined based on how likely each factor would be to cause detrimental ripple effects across the entire gesture vocabulary, similar to what is recommended for factors that are interrelated. Factors that, when considered, would require modifications to a number of gestures or the entire vocabulary, should have higher priority and thus be evaluated before those that would require minimal modifications. For example, a factor such as Social Acceptability could require that an entirely new gesture be created to conform to cultural or location norms. In comparison, there are user interface changes that could be implemented to decrease the Occlusion of a gesture. Therefore, within the context of this example, Occlusion should be considered later in the process than Social Acceptability.
A challenge of any iterative design process is to determine when to stop. Most often, an endpoint is reached when one runs out of time or resources, however, achieving the perfect gesture vocabulary may never be possible as the design process itself is inherently one large set of tradeoffs. After iterative evaluation and refinement, the resulting gesture vocabulary should fulfill the requirements of the preconditions and the factors that were deemed to be high importance because important factors should have received enough revision, and been revised early enough, during the process.
5.1.3 Summary.
Although this process is but an example, we believe that the creation and use of user-centered, factor-centric approaches have the potential to ensure that the gesture vocabularies that are designed will be the most usable, safe, and beneficial for their target populations. Different from existing gesture design methodologies, which focus on designing suitable gestures by leveraging the expertise, experience, and capability of experts, users, or systems individually, taking a user-centered, factor-centric approach to gesture design should enable the benefits of each different methodology to be used, when appropriate.
Future work is obviously needed to validate this new user-centered, factor-centric gesture design approach by comparing it with existing methodologies, not only to ensure that the resulting gesture approach results in more appropriate gesture sets, but also to evaluate the costs associated with their use (e.g., number of iterative cycles needed, monetary cost of running multiple user studies, etc.) and gain insights into reducing the cost of iterations. If for nothing else, we hope that the proposal and outline of such an approach will improve awareness about the challenges inherent in the design of gestures, the dependencies that exist between different factors, and the potential pitfalls of using the most popular or easy-to-use method to create gesture vocabularies.
5.2 The Evolving Ecosystem of Gesture Design Factors
Although the literature review and analysis identified 13 factors that are critical for the design of gestures, it may never be possible to identify and distill the complete set of factors that influence how users remember, learn, and perform gestures, and how systems can support them in these tasks. We first discuss the role of factor interdependencies on gesture design, detail some additional, higher-level factors that emerged, and then provide commentary on the “naturalness” of gestures.
Understanding the complex connections among various factors is essential for gesture design, as it enables designers to better understand the implications of prioritizing different factors and resolving tradeoffs when designing gesture vocabularies. The identification of the 13 factors was a first step toward constructing a comprehensive picture of gestural interaction. However, much future work is needed to better understand how to disentangle factors such as Complexity and Efficiency, Intuitiveness and Learnability, Ergonomics and Recognition, Occlusion and Feedback, and so on, as well as if and how they should be designed for independently. In addition to the identified factors, it is also possible that new primitive factors may arise and should be added to the ecosystem of factors as the field furthers its understanding of gesture design. While the user-focused, factor-centric approach that was proposed is extensible and should allow for the addition of new factors, care must be taken to identify and understand the relationships that exist between the newly discovered and existing factors. To encourage the inclusion of new factors into the ecosystem, we plan to construct an online, accessible, and community-maintained database and visualizations of the factors and relevant research articles about these factors to ensure the ecosystem can stay relevant and will evolve with time.
During the literature review, we encountered three other factors that are not included in the final set of 13 factors. The first factor that emerged was the accessibility of gestures, e.g., for older adults [
179,
253], as well as with users who have motor [
388], or visual impairments [
142,
143,
145]. Within the factors that we identified, we did not include accessibility as a distinct factor, because accessibility is a higher-level factor that can be decomposed into a number of the factors that we identified, such as Context, Complexity, Learnability, and Feedback. For example, accessibility is related to situational factors such as Context, because the requirements of users’ accessibility need to be understood and considered by designers. It is also related to cognitive and physical factors such as Complexity and Learnability, because a user’s cognitive and physical abilities may pose different design constraints on these factors [
179,
253]. Accessibility is also be associated with system factors, such as Feedback, because tactile instead of visual feedback may need to be evaluated for users with visual impairments [
143]. As this factor, and likely others, can be decomposed into a subset of the 13 factors that this present research identified, it seems important to first establish the set of primitive factors before the inclusion of higher-level factors.
The second factor was multi-user gestural interaction (i.e., the use of gestural interaction in multi-user collaborative or cooperative settings [
174,
215,
216]). The factor identification and analysis provided within this article was limited to single user use cases because, as the analysis began to demonstrate, there was yet to be a consensus about all of the factors implicated in the design of single user gestures, let alone a methodology that could evaluate all of the factors at the same time. As the multi-user use case is a more complex version of the single-user scenarios that are commonly evaluated and used as use cases in the literature, it did not seem appropriate to dive into this more complex topic before a solid foundation of single user gestural interaction was developed. While the findings about some factors such as Social Acceptability and Context may seem, at first glance to be the most applicable to these scenarios, there is much need and opportunity to explore how
all the 13 factors influence these scenarios and also to identify any additional factors that could be unique to multi-user settings (e.g., multi-user cooperation, user roles, territoriality, and so on).
Lastly, one of the most popular factors that emerged was
naturalness, which has also been referred to as
fluidity,
feelings,
satisfaction,
controllability,
affect, and
familiarity.
14 Naturalness, however, is complex factor to consider because it is difficult to define and measure objectively. Baudel and Beaudouin-Lafon argued that gestural interaction is
natural because it builds upon the manipulation and gestural communication skills that humans acquire naturally [
37]. Wigdor and Wixon, however, proposed that naturalness is not in fact a design factor at all, but rather an experience goal, wherein a system should evoke a feeling of effortlessness and enjoyment that results in users “act[ing] and feel[ing] like a natural” in its use [
357]. These “natural” feelings are often compounded by many relevant factors, such as Intuitiveness, Learnability, and Ergonomics. The pinch to zoom gesture, for example, does not mimic or build upon an existing hand motions or metaphors that users are accustomed to, but over repeated presentations and performances of the gesture, has been described as a fluid, satisfying, familiar gesture by users who perform it everyday [
357].
Gestural interaction has often been used as one example of a “natural” user interface when compared to the Windows, Icons, Menus, Pointer user interface metaphor [
320]. Despite naturalness having been repeatedly referred to in the literature, the definition of naturalness has been inconsistent [
37,
357] and was often used alongside descriptions of other factors such as Learnability, Intuitiveness, and Complexity [
119,
121,
359,
376]. Given the lack of consensus regarding what constitutes a “natural” movement or feeling and the fact that naturalness is often referred to as an experience goal rather than as a design factor [
101,
208,
300], naturalness was not considered to be one of the factors crucial to the design of gestures within the context of this work. We respectfully encourage the community to avoid the use of the terms “natural” or “naturalness” when describing interactive systems, and instead focus on using terms to describe the situational, cognitive, physical, and system factors that govern gestural interaction, such as those that were identified, analyzed, summarized in this work.