Keywords

1 Introduction

Several input systems using eye blinking have been proposed [1–7], one of the purposes of which is communication for the severely disabled. Eye blinking is either voluntary or involuntary [8]. A system employing a human-computer interface based on eye-blink information is used to distinguish a user’s input requests based on the classification of voluntary blinks. For application to human-computer interfaces, we previously proposed an automatic classification method for both involuntary blinking and one type of voluntary blinking [9]. However, if the types of classifiable voluntary blinking increase, we can assign an individual command to each type. Applying this to a human-computer interface will significantly improve the efficiency when inputting commands.

In previous studies, several measurement methods using an image analysis for eye blinking were proposed. Based on these measurement methods, several classification methods for voluntary blinking using a special pattern of eye blinks were proposed [1–5]. As a special pattern of eye blinks, some systems employ a blinking duration of several seconds [1–4]. For one of these systems, multiple levels of thresholds in duration were set to distinguish additional input requests in addition to standard inputs [4]. A classification system for voluntary blinking using multiple blinks was also proposed [5]. In this system, a sequence of multiple blinks is assigned to a command that emulates a mouse click. By performing many more voluntary blinks, users can input a multiple click command along with standard click commands into the system. However, employing such special blinking types may be a burden to users.

Previously, we developed a classification method for both involuntary blinking and one type of voluntary blinking, i.e., blinking firmly. The purpose of this paper is to classify a new type of voluntary blinking in addition to our proposed classification between involuntary blinking and one type of voluntary blinking. To achieve this purpose, we introduce a new blinking parameter. Based on this parameter, we also propose a new classification method for involuntary blinking and the two types of voluntary blinking.

2 Eye Blinking Measurement Based on an Image Analysis

Several measurement methods for eye blinking have been traditionally proposed. One of these measurement methods is the electro-oculogram (EOG) [6], [8], which measures the difference in potential between the retina and cornea from electrodes placed near the user’s eye. On the other hand, a detection method using an infrared sensor is proposed [7]. In a system employing this method, infrared LEDs or other lighting equipment emit light to the user’s eyelid. A variation of the reflected light is measured using an infrared sensor. These measurement methods require some specific equipment for eye-blink detection. Hence, we employ a method based on an image analysis for measuring eye blinking [9]. This method can be set up with a PC and a video camera for home use.

Typically, a high-speed video camera has been required for measuring eye blinking in detail [10]. However, we developed a method for measuring eye blinking using field images from interlaced images [9], which we refer to as a “frame-splitting method.” This developed method yields a time resolution double that of a standard NTSC or 1080i Hi-Vision camera, i.e., 60 fps. Using field images split through this method, our measurement system calculates the pixels of an open-eye area as a measurement value. A detailed eye-blinking wave pattern is then obtained by recording these values in a time series.

We then employ an automatic extraction method using statistical parameters calculated from this pattern [9]. The thresholds for this extraction are determined by the following equations:

$$ {\text{Th}}_{1} = {\overline{\text{E}}} + 2\sigma $$
(1)
$$ {\text{Th}}_{2} = {\overline{\text{E}}} - 2\sigma , $$
(2)

where \( {\overline{\text{E}}} \) is the average difference value when the eye is open, and \( \sigma \) (sigma) is its standard deviation. The difference value is defined as the difference in measurement values yielded from a sample point and the immediately previous point. If the difference value is larger than \( {\text{Th}}_{1} \), the sample point indicates that the eye is opening. Similarly, if the difference value is smaller than \( {\text{Th}}_{2} \), the sample point indicates that the eye is closing. The part between the closing sample points and the opening sample points is extracted as a wave pattern of an eye blink.

3 Classification of Blink Types

Previously, our classification method was able to classify blinking types between involuntary blinking and one type of voluntary blinking when performed firmly. To increase the classifiable type of voluntary blinking, we need to introduce a new type of voluntary blinking. In this chapter, we introduce two types of voluntary blinking in which a new blinking type is added. We also show a new parameter for classifying these types of eye blinks.

3.1 Two Types of Voluntary Blinking

In addition to standard voluntary blinking, blinks of several seconds in duration, as well as multiple blinks, can be employed as additional blinking types. However, a system using blinking of several seconds in duration can make users feel that their time is being wasted [5]. This type of voluntary blinking may burden users. If multiple blinks are employed as additional voluntary blinking, the user needs to count the number of voluntary blinks while performing voluntary blinking. This parallel operation may place a cognitive burden on users more than simple blinking, which may cause user fatigue. Psychological burden, such as fatigue, may affect the wave patterns of eye blinks [8]. Thus, our new classification method employs two types of voluntary blinking, i.e., “blinking firmly” and “blinking firmly for a brief time.” Figure 1 shows the wave patterns of eye blinks measured during our experiments.

Fig. 1.
figure 1

Wave patterns of involuntary blinking and two types of voluntary blinking

Full size image

The x-axis in Fig. 1 indicates the sampling point, and the y-axis indicates the open-eye area pixels. These plots were normalized based on the estimated value of the first field image. The black line is a wave pattern from blinking firmly, and the gray line is a wave pattern from blinking firmly for a brief time. Finally, the dashed line is a wave pattern for involuntary blinking. From Fig. 1, we can see that the difference in duration between blinking firmly, which we proposed previously, and blinking firmly for a brief time, is significant. However, the difference in duration between blinking firmly for a brief time and involuntary blinking may be small.

In our previous studies, we used the duration of eye blinking as a parameter for classifying between involuntary blinking and blinking firmly, which is one type of voluntary blinking. However, from Fig. 1, we can estimate that the duration for blinking firmly for a brief time and that for involuntary blinking are similar. Thus, classifying these three types of eye blinks with a high degree of accuracy using only this parameter is difficult to achieve. We therefore propose an integrated value of the amplitude as a new parameter for this type of classification.

3.2 New Parameter for Classification of Blinking Types

Previously, we investigated the feature values of an eye blink based on employing the duration and maximum amplitude as parameters. Figure 2 shows the feature parameters of an eye blink.

Fig. 2.
figure 2

Wave pattern parameter definitions

Full size image

The duration is defined as the period of time between the start point and end point of an extracted wave pattern. The maximum amplitude is defined as the difference in amplitude between the measurement value of the start point and that of the point that changes the most from the start point. In this paper, we propose a new parameter, i.e., an integrated value (summation) of the amplitude, in addition to the parameters used in our previous studies. We define the summation of the difference between the baseline and measurement values as an integrated value of the amplitude, which is shown in the shaded region in Fig. 2. The baseline for each extracted wave pattern is set individually as the smaller of the start and end points. This new parameter is calculated based on the time integration (summation) of the amplitude from the start point to the end point. We expect that this new parameter, the integrated value of the amplitude, will show significant differences among the three types of eye blinks.

4 Metering Experiments

To extract two types of voluntary blinks, we conducted two 30-s long experiments. Before the start of each experiment, the subjects were asked to keep their eyes open for 3 s to estimate the thresholds, \( {\text{Th}}_{1} \) and \( {\text{Th}}_{2} \), of our extraction method, which utilizes Eqs. (1) and (2) described in chapter 2. During the measurements, beeping sounds were generated at random intervals of 5 to 7 s. For experiment 1, we detected their voluntary blinks after asking the subjects to close their eyes firmly upon hearing a beep. For experiment 2, we instructed them to briefly close their eyes firmly. We also detected their involuntary blinks through the same experiments.

Our measurement system includes a Hi-vision video camera (Sony HDR-HC9, for home use) and a PC (OS, Microsoft Windows 7; CPU, 2.8 GHz Intel(R) CoreTM i7). In each of our experiments, an image sequence of each subject’s eye blinking was recorded and stored on the PC. We used the PC to analyze these recorded image sequences. In this chapter, we describe the measurement results for the parameters obtained from successfully extracted blinks using Eqs. (1) and (2). We also discuss the measurement results.

4.1 Measurement Results of Parameters

Figure 3 shows the measurement results for the eye-blinking duration. In Fig. 3, the y-axis indicates the duration obtained from each type of eye blinking. The bar graphs indicate the average values for blinking firmly, blinking firmly for a brief time, and involuntary blinking in order, from the left. Figure 3 makes it clear that the difference between blinking firmly and involuntary blinking was significant for each subject. Similarly, the difference between blinking firmly and blinking firmly for a brief time is also large. However, the difference between blinking firmly for a brief time and involuntary blinking was slight for several of the subjects, including subjects A and G.

Fig. 3.
figure 3

Measurement results of the duration

Full size image

Figure 4 shows the measurement results for the maximum amplitude parameter. In Fig. 4, the y-axis indicates the normalized maximum amplitude of an eye blink. The bar graphs in Fig. 4 indicate the average values of this parameter, which were obtained from each type of eye blinking in the same way as shown in Fig. 3.

Fig. 4.
figure 4

Measurement results of the maximum amplitude

Full size image

In the comparison of this parameter, the difference between the two types of voluntary blinking is slight, and the difference between each type of voluntary blinking and involuntary blinking is significant for many of the subjects. However, the differences among the three blinking types are slight in certain cases, such as subject E. Additionally, there are significant individual differences in the relations based on the magnitudes of this parameter when obtained from the three types of blinking.

Figure 5 shows the measurement results based on the integrated value of the amplitude. In Fig. 5, the y-axis indicates this parameter, and the bar graphs indicate the average values of the parameter obtained from each type of eye blinking in the same way as in Fig. 3.

Fig. 5.
figure 5

Measurement results of the integrated value of the amplitude

Full size image

The results shown in Fig. 5 make it clear that blinking firmly has the largest integrated value of the amplitude, followed by blinking firmly for a brief time, and involuntary blinking. In addition, the differences between each blinking type are significant. Based on this tendency, we propose a new classification method between involuntary blinking and the two types of voluntary blinking.

4.2 Automatic Classification Between Involuntary Blinking and Two Types of Voluntary Blinking

The integrated value of the amplitude tends to be largest for blinking firmly, followed by blinking firmly for a brief time and involuntary blinking. A new classification method based on this tendency enables us to classify both involuntary blinking and the two types of voluntary blinking. The thresholds for the classification of these three blinking types are determined through the following equations:

$$ {\text{Th}}_{ia1} = \frac{{\overline{{{\text{Ia}}_{v1} }} + \overline{{{\text{Ia}}_{v2} }} }}{2} $$
(3)
$$ {\text{Th}}_{ia2} = \frac{{\overline{{{\text{Ia}}_{v2} }} + \overline{{{\text{Ia}}_{iv} }} }}{2} ,$$
(4)

where \( \overline{{{\text{Ia}}_{v1} }} \), \( \overline{{{\text{Ia}}_{v2} }} \), and \( \overline{{{\text{Ia}}_{iv} }} \) are the average integrated values of the amplitude obtained from blinking firmly, blinking firmly for a brief time, and involuntary blinking, respectively.

Our new classification method compares each threshold and the integrated value of the amplitude obtained from the extracted blinks. If the value of this parameter is larger than threshold \( {\text{Th}}_{ia1} \), the blinking type is classified as blinking firmly. Otherwise, if the value is larger than threshold \( {\text{Th}}_{ia2} \), the blinking type is classified as blinking firmly for a brief time. Finally, if the value is not larger than threshold \( {\text{Th}}_{ia2} \), the blinking type is classified as involuntary blinking. These thresholds were determined for each of the subjects and using the same experimental data as the estimation of the classification rates. The classification results of the blinking types using our proposing method are shown in Table 1.

Table 1. Classification results for involuntary blinking and two types of voluntary blinking
Full size table

Table 1 shows the number of extracted blinks that were classified into the three blinking types for each subject. This table also shows the number of classification errors that occurred during these experiments using our proposing method. Table 2 shows the results of the classification rates calculated using the numbers of classified blinks and classification errors listed in Table 1.

Table 2. Classification rates between involuntary blinking and two types of voluntary blinking [%]
Full size table

The results of the total classification rate in Table 2 show that the average rate of successful classification is approximately 95 % for the experimental sample of seven subjects using our new method for classification between involuntary blinking and the two types of voluntary blinking. Previously, several methods for classification between long and short voluntary blinks were proposed [1], [3]. These methods employed an image analysis, which calculates the correlation coefficients of the template matching as an eye-blinking wave pattern. This definition of the wave pattern is different from that used in our measurement method. However, these methods classify the blinking types based on the duration parameter, which is the same as in our method. Grauman et al. conducted an experiment to investigate the classification rate using long and short voluntary blinks for 15 subjects [1]. The results of this experiment show a successful classification rate of 93.0 %. On the other hand, Królak et al. reported a classification rate of 98.31 % [3] using these types of voluntary blinking. However, in each experiment described in these studies, a short voluntary blink and an involuntary blink are treated as equal. In other words, these types of blinks are not classified. Our experimental results show similar classification rates as these previous experiments. In addition, our experimental results also show that our proposed method can properly classify between involuntary blinking and the two types of voluntary blinking.

5 Conclusions

In previous studies, we developed a measurement method using an image analysis to yield variations in an open-eye area as a wave pattern. This measurement method enables us to extract wave patterns of eye blinks. Using this measurement method, our experimental system can classify between involuntary blinking and one type of voluntary blinking, i.e., blinking firmly.

In this paper, we added a new type of voluntary blinking, and measured three types of eye blinking: blinking firmly, blinking firmly for a brief time, and involuntary blinking. For classifying these blinking types with a high degree of accuracy, we introduced a new blinking parameter, i.e., the integrated value of the amplitude. This parameter was compared for each blinking type, and other parameters that we previously proposed, i.e., the duration and maximum amplitude, were also compared. Based on the results obtained from this comparison, we proposed a new method for classifying these three blinking types. Using our proposed method, we yielded an average classification rate of approximately 95 % for seven subjects.

In the future, we plan to investigate the efficiency of our proposed method experimentally using additional subjects. The classification method proposed in this paper was designed on the basis of a single parameter, the integrated value of the amplitude. To yield a higher degree of accuracy, we also plan to consider a better method using the multiple parameters described herein.