Keywords

1 Introduction

Conventionally two-dimensional GUI is operated by a finger, a stylus on a touch-screen of a tablet PC, or a mouse on a PC. Recently the operation by using hand gesture in three-dimensional space is proposed as the new method of the operation of two-dimensional GUI. It has some advantages that the user can intuitively operate two-dimensional GUI by using a physical motion, and also operate the GUI from a distance without any device. Fitts’ law exists as a model of operation of two-dimensional GUI which is pointing with a mouse.

$$ {\mathbf{MT}} = {\mathbf{a}} + {\mathbf{b}} \cdot {\mathbf{log}}_{{\bf 2}} \left( {\frac{\varvec{W}}{\varvec{D}} + {\bf 1}} \right) $$
$$ {\mathbf{ID}} = {\mathbf{log}}_{{\bf 2}} \left( {\frac{\varvec{W}}{\varvec{D}} + {\bf 1}} \right) $$

In previous studies [1,2,3,4], it was clarified that the expanded Fitts’ law to the two-dimensional GUI operation was applicable to the time required for the operation by the mouse to move the cursor to the target and click the target on a screen (mouse pointing operation). As for the pointing operation by the hand gesture (gesture pointing operation), it was suggested that the expanded Fitts’ law did not necessarily apply to the operation time and that the target size only had a minor influence on the operation time [3]. This research focused on examining the cause the expanded Fitts’ law could not apply to the operation of two-dimensional GUI using hand gestures through two experiments.

2 Method

2.1 Participants

Seventeen right–handed university students ranging in age from 20 to 24 who volunteered and were not provided any payment, took part in the experiments. Twelve participants took part in Experiment 1 and five participants took part in Experiment 2. They received a thorough explanation about the method of the experiments and signed the consent form.

2.2 Experiment Tasks

Experiment 1 examined the characteristics of the mouse pointing operation and the gesture pointing operation. Experiment 2 examined the characteristic of the gesture operation by two kinds of the CD (Control-Display) ratio (see Sect. 3.1). The participants were required to do the multi directional pointing task (Fig. 1) which was modified ISO9241-411 Annex B [5] for using hand gesture on the 58-inch large screen. The index of difficulty (ID) has 9 patterns as shown in Table 1 and the method of calculating ID is based on the extended Fitts’ Law (two-dimensional model). They were required to place the cursor over the target when it turned red and hold it there for two seconds until the next target turned red.

Fig. 1.
figure 1

Experimental tasks

Full size image
Table 1. Patterns of Index of Difficulty
Full size table

2.3 Experiment Environment and Apparatus

For Experiment 1, the gesture pointing operation was done with the large screen (58-inch, 1920px × 1080px). The mouse pointing operation was done with the small screen (21.5-inch, 1920px × 1080px). The visual distance was set to 2 m and 0.7 m respectively. For Experiment 2, the gesture pointing operation was done with a large screen (58-inch, 1920px × 1080px). The visual distance was set to 2 m. In both experiments, the motion capture system used VEANUS 3D, Nobby Tech. Ltd. Also, all the participants required to pick up a reflective material marker by the thumb and index finger of the right hand during the experiment.

The time from changing the color of the target to putting the cursor on the target was measured as the pointing time. For the data obtained, Smirnov-Grubbs test was conducted at a significance level of 0.05 in order to remove the unexpected times values.

3 Results and Discussions

3.1 Experiment 1

Figure 2 showed that the relation between the ID and the pointing time. The solid line and dotted line showed the results of the simple linear regression analysis.

Fig. 2.
figure 2

Relation between the ID and the pointing time

Full size image

The regression expression of the mouse pointing operation was

$$ \varvec{MT} = {\mathbf{341}}{\mathbf{.61}} + {\mathbf{169}}{\mathbf{.11}} \times \varvec{ID} \quad \varvec{R}^{{\mathbf{2}}} = {\mathbf{0}}{\mathbf{.9579}} $$

The expression of gesture operation was

$$ \varvec{MT} = {\mathbf{1335}}{\mathbf{.5}} + {\mathbf{240}}{\mathbf{.39}} \times \varvec{ID}\quad \varvec{R}^{{\mathbf{2}}} = {\mathbf{0}}{\mathbf{.262}} $$

The results of the regression analysis revealed that mouse operation applied to Fitts’ law and that the gesture operation did not necessarily apply to Fitts’ law.

Figure 3 showed that the example of velocity waves of the mouse pointing and the gesture pointing. Both velocity waves of the mouse operation and the gesture operation were convex upward. The waveform of the mouse operation was not monotonously decelerated but there were both the decelerated section and the constant section after the maximum speed was reached. On the other hand, the waveform of the gesture operation monotonically decelerates after the maximum speed was reached. The maximum speed and the appearance time of the maximum speed are different between the mouse operation and the gesture operation. Asahi [6] pointed out that the maximum speed in the mouse operation and its appearance time are influenced by the CD ratio, and that the smaller the CD ratio was, the higher the maximum speed was and the faster the appearance time was. The velocity waveform of the gesture operation in this experiment was very similar to the waveform of the mouse operation with the large CD ratio in the previous research [6]. In addition, the CD ratio of the gesture operation in this experiment is larger than the large CD ratio of the previous research. From these relations between the velocity waveform and the CD ratio, it was suggested that the difference of the velocity waveform between the mouse operation and the gesture operation might be influenced by the CD ratio. Therefore, we set two types of CD ratios and conducted Experiment 2 with the gesture operation. One was the conventional CD ratio for the gesture operation (CD ratio = 1), and the other was the CD ratio recommended for the mouse operation (CD ratio = 0.625).

Fig. 3.
figure 3

Velocity waves of the mouse pointing and the gesture pointing

Full size image

3.2 Experiment 2

Figure 4 showed that relation between the ID and the pointing time. The solid line and dotted line showed the results of the simple linear regression analysis.

Fig. 4.
figure 4

Relation between the ID and the pointing time

Full size image

The regression expression of gesture operation with the conventional CD ratio was

$$ \varvec{MT} = {\mathbf{1476}}{\mathbf{.5}} + {\mathbf{21248}} \times \varvec{ID}\quad \varvec{R}^{{\mathbf{2}}} = {\mathbf{0}}{\mathbf{.2634}} $$

The expression of gesture operation with recommended for the mouse CD ratio was

$$ \varvec{MT} = {\mathbf{973}}{\mathbf{.82}} + {\mathbf{205}}{\mathbf{.81}} \times \varvec{ID}\quad \varvec{R}^{{\mathbf{2}}} = {\mathbf{0}}{\mathbf{.5124}} $$

The results of this experiment revealed that the contribution rate of the gesture operation with the recommended CD ratio increased in comparison with the gesture operation with the conventional CD ratio. The results were suggested that the differences in the degree of the appliance to Fitts’ law depend on the difference of the CD ratio. However, the gesture operation with the recommended ratio did not satisfyingly apply to Fitts’ law in comparison with the mouse operation with the same CD ratio. The small number of the participants might be why the contribution rate was not highly satisfying. In the future, it will be necessary to collect a larger number of the data set and analyze it.

4 Conclusion

This research explored the characteristics of the hand gesture pointing operation through two experiments. In the first experiment (Experiment 1), by comparing the velocity waves of the mouse operation with the gesture operation, we found the difference between them and it was suggested that this difference might be due to the Control-Display ratio. In the second experiment (Experiment 2) it was examined whether the contribution rate of the regression analysis for the pointing time of the gesture operation depended on the CD ratio by two kinds of the CD ratio (CD ratio = 1 as the conventional and CD ratio = 0.625 as the recommended for the mouse operation). The results of the experiment revealed that the contribution rate of the gesture operation with the recommended CD ratio increased in comparison with the conventional CD ratio. However, the gesture operation with the recommended ratio did not satisfyingly apply to Fitts’ law in comparison with the mouse operation with the same CD ratio.

In conclusion, the results of the experiments suggested that the velocity waveform of the pointing time of the mouse operation was different from the gesture operation and the difference of the waveform might be influenced by the CD ratio and that the difference in the degree of the application to Fitts’ law depended on the difference of the CD ratio. However, the gesture operation with the recommended ratio did not satisfyingly apply to Fitts’ law in comparison with the mouse operation with the same CD ratio. In the future, it will be necessary to collect a larger number of the data set and analyze it in order to examine why the contribution rate was not highly satisfying.