1 Introduction

Gesturing is a natural phenomenon that is found even in individuals who are congenital blind [1]. Interaction using gesture is becoming more and more popular day by day. It provides a more natural and comfortable means of interaction. Recent advancement in gesture recognition algorithms and low-cost hardware development had made it possible to use gesture commands to interact with computer and other consumer electronics. Despite active research in gesture-based interaction and related user-elicited studies, much attention is needed to include visually impaired users [2]. It is equally important for them to interact with computers. The sighted user provide input using mouse and keyboard. However, visually impaired users find it difficult to use them effectively.

Keyboards like e-Brailler, SMART Brailler, etc. are available in the market to assist them. However, facts reveal that only 8–9% of visually impaired students in the United States use Braille [3]. The majority of their population (\(\sim \)90%) lives in developing countries where the Braille literacy rate is as low as 3%. Hence, solutions based on Braille are not so popular for computer interaction [4]. Apart from Braille, interaction using data gloves is also proposed in the literature. But the data gloves approach needs the user to wear a special device which hinders the naturalness [5]. Speech processing is ineffective for the mentioned purpose as it depends on accents, dialects, and mannerisms [6]. Even, systems based on EEG are unsuitable because the received information is noisy and ambiguous [7]. Handwriting recognition via smartpens seems to be a potential technique. However, the letters like f, i, j, t, x, consists of two strokes wherein the second stroke is referential to the first stroke (dotting the j’s and crossing the f’s). Further, locating precise spots on the touch screen surface can be very difficult for visually impaired. Research [8] confirms that visually impaired users can draw and make some gestures on touchscreens with more or less difficulty. They face severe issues with form closure, line steadiness, location accuracy while drawing gestures [9]. Consequently, it is complicated for them to learn and use handwriting [10].

Fingerspelling in American Sign Language (ASL) or its subset have been used extensively in many HCI applications. However, these fingerspelling signs were devised with deaf and mute users. Visually impaired users do not know sign language. Additionally, they feel difficult to use it. Past studies [11] have suggested that including the target users in the gesture elicitation study will increase the usability of the system than just using gestures of ASL. Hence, a study [2] is performed with visually impaired users. Based on which a dactylology [12] is proposed for them to interact with computers. Work in [12] primarily focuses on the recognition aspect of the dactylology.

Before we proceed further, it is important to clarify the motive behind this work and its difference with that of the [2]. The motive of this work is to fill the knowledge gap and provide insight on gesture elicitation study with visually impaired users. Some of the key points are listed below.

  • In this work, we explicitly answer following questions about the gesture elicitation study.

    • What design aspects/criterion are considered for the selection of optimal gestures?

    • How & why these set of gestures are further categorized into two-tier?

    • Which gestures are mapped to the most commonly used keys of the computer keyboard and why?

    • How is cognitive load reduced?

  • Detailing of ergonomics aspects which were considered while performing the gesture elicitation study.

  • Limitations and issues faced while the gesture elicitation study is also discussed.

The remainder of the article is organized as follows. Section 2 discusses the gesture elicitation study. Section 3 presents a glimpse of dactylology. Section 4 presents the result and discussion while Sect. 5 presents conclusion and future work.

2 Gesture Elicitation Study

The sense of touch in a visually impaired is stronger than normal vision. Hence, a tabletop set-up is provided in the proposed system. This set-up facilitates haptic feedback and support to the arms. One of the vital questions is what type of gesture should be used. Previous research shows that finger-based gestures cause less fatigue and are more comfortable as compared to arm-based gestures [13]. Visually impaired also find finger-based gestures are easy. Hence, only finger-based gestures are investigated in this work.

A finger is considered to be either in extension- or flexion- state. In extension state, the particular finger is stretch out, while the flexion state involves folding of the finger. We have assigned a number to each finger, i.e. thumb \(\rightarrow \) 1, index \(\rightarrow \) 2, middle \(\rightarrow \) 3, ring \(\rightarrow \) 4, little \(\rightarrow \) 5. The naming convention of gesture used in the study consist of letter G that resembles gesture followed by the number of finger(s) in the extended state, e.g. G15, G2345, etc. G15 means a gesture with the thumb (1) and little (5) fingers in the extended state. Similarly, G2345 means gesture with index (2), middle (3), ring (4), and little (5) fingers in the extended state.

Fig. 1.
figure 1

Illustration showing correct hand posing based on ergonomics.

Full size image
Fig. 2.
figure 2

Illustration of gesture selection method.

Full size image

With the help of 5 fingers, 31 possible gestures can be formed. These gestures are divided into 5 classes. Here, a class is the set of gestures with an equal number of extended fingers. It is found that not all the possible 31 gestures are comfortable to the user. The physiological constraints and interrelation between joints of the finger cause fatigue. It is crucial to find gestures that are optimal for visually impaired users. Therefore, we performed a gesture elicitation study with 25 visually impaired participants. These participants were graduate students (avg. age \(\simeq \)22) with no prior gesture posing experience. While acquiring gestures, participants were asked to settle down themselves in a relaxed position. We have ensured that participants keep their forearm flexed with elbow at \(90^{\circ }\). Additionally, there should not be any bent in the wrist as shown in Fig. 1.

The forearms and wrists should be kept in-line with the shoulder to provide a comfortable experience to the participants.

Two important metrics—performance & preference—are considered to choose optimal gesture. Illustration of gesture selection method is shown in Fig. 2. In the first stage, optimal gestures are obtained on the basis of the performance metric. In performance metric, a user is asked to form a gesture and evaluate it on four subjective criteria defined as below.

  • Easiness: A parameter to figure out the fatigue [14] while executing/posing hand gestures.

  • Naturalness: It is a parameter which considers the likeness of the gesture being used in natural everyday human behaviour [15].

  • Learning: Through this [14] criterion, we try to figure out whether a gesture can be learned and adopted.

  • Reproducibility: A parameter to measure the reproduction ease of a gesture. We considered fist to be neutral pose and asked participants to repeatedly (4 times) produce the gesture from fist and finally, rate the gesture based upon its reproduction easiness.

Fig. 3.
figure 3

Performance and preference metric (a) performance metric (b) preference index.

Full size image

The rating is done on a scale of 1–5 using a Likert scale. The overall rating of a gesture is calculated by finding the sum of all the criteria. Performance metric is obtained as the median of the overall rating. This is depicted as bar plot in Fig. 3(a). Gestures with the performance metric greater than 32 are considered to be optimal. Among the possible 31 gestures G1, G2, G5, G12, G15, G23, G25, G123, G125, G234, G345, G2345 and G12345 are found to be optimal. In preference metric, users are requested to pose gestures and ranked them based on their preferences. It should be noted that gestures are ranked on an ordinal scale within its class only. A preference index \(P_{i}\) is calculated using these preference ranking whose result is furnished in Fig. 3(b).

The above mentioned optimal gestures are further categorized into two grades: tier-1 and tier-2 gestures. Each class has a gesture with the highest preference index which is included in tier-1 grade. Let us consider a class-2 case shown in Fig. 2. It can be observed from Fig. 3(b) that gesture G23 has the highest \(P_{i}\) among class-2 gestures. Rest gestures of class-2 (i.e. G12, G23 & G15) are included in tier-2 gestures. Similarly, analyzing other class, gesture G2, G23, G234, G2345, and G12345 are found to be categorized in tier-1 grade. Rest of the optimal gestures are considered as tier-2 grade gestures.

3 Dactylology

The dactylology [12] uses tier-1 gestures to map alphabets and numbers. It uses a combination of left and right hand to produce alphabets while single hand (i.e. either left or right) is used to produce numbers. The combination is formed according to the Table 1 as depicted in Fig. 4. Character A in the matrix is an element of the first row and the first column. The symbol for character A is formed by the combination of the left hand with one finger and right hand with one finger. Similarly, other symbols for remaining alphabets and numbers can be formed as shown in Fig. 4. The word is formed by concatenating the subsequent symbols. The dactylology illustrated in Fig. 4 is reproduced for the completeness and understanding of the article. For additional details, please refer to [12].

Fig. 4.
figure 4

(©2017 Kishor Prabhakar Sarawadekar and Gourav Modanwal. All rights reserved)

Left: illustration of the matrix. Right: dactylology (a) Symbols for alphabet, (b) symbols for number.

Full size image

4 Result and Discussion

The primary motive of this work is to provide detailed insight on gesture selection methodology considering visually impaired users. The study is done with 25 visually impaired participants. In this work four subjective criteria: easiness, naturalness, learning, and reproducibility are used. Users rated each gesture on a scale of 1–5 using a Likert scale. Performance metric of each gesture (refer Fig. 3(a)) is obtained from the user’s responses. This metric indirectly tells about the goodness factor of the gesture. Based on the performance metric, an optimal gesture set is obtained.

The obtained optimal gestures of each class contain a gesture which is better than the others. In order to obtain the best gesture of its class, we calculated the preference index (refer Fig. 3(b)). Now the obtained optimal gesture set is categorized into two-tier grade. Tier-1 gestures are those gesture which has the best preference index among its class. Rest are Tier-2 gestures. The benefit of having two-tier gesture among the optimal gesture facilitate the option of mapping the most commonly used keys of the keyboard to tier-1 gesture. These are the most preferred gesture among the obtained optimal gesture set. More than 200 symbols can be created using tier-1 and tier-2 gestures. However, these symbols are required to be mapped to keys of the keyboard through a gesture command matching study.

Cognitive load is another vital parameter in the development of gesture set for a large set of task. Since a participant can remember \(5\pm 2\) gestures in short-term memory [16]. Hence, the number and type of gesture must be carefully chosen. The aforementioned cognitive load is reduced by reusing similar gesture under a different context. The reuse is done on the basis of the matrix shown in Fig. 4. This reuse of gesture will not only reduce cognitive load but also facilitate a larger set of the tasks with a smaller number of gestures.

5 Limitations/Issues Faced

We faced some issues during data collection. Few participants were unable to complete the study in one session. The remaining part of the study was done on the subsequent day. This may affect the ratings by the users. However, we have ignored this factor in the present study.

All the participants were male with average age \(\simeq \)22 years. School children and female participants were not considered in this study as there were issues while obtaining ethical approvals for them. It will be important to understand their view on an optimal gesture selection.

When analysing the questionnaires response, we came across the problem of treating Likert scale as either ordinal or interval. It is a long-running dispute. We have treated it as ordinal scale and computed the median of the overall score obtained as the sum of all criteria.

6 Conclusion and Future Work

In this work, we present insight and discussion on the gesture selection method. Based on the outcome of the quantitative rating analysis with 25 visually impaired participants, an optimal gesture set is devised. Further, a novel dactylology is proposed for them to interact with computers. Real-time experiments are also performed with visually impaired users, and motivating results are obtained. Clearly, further research work such as mapping of remaining tier-2 gestures, finding the effect of handedness, analyzing cognitive load, comparison with existing methods, etc. are required.