Multimodal human eye blink recognition method using feature level fusion for exigency detection

Abstract

In this paper, a precise multimodal eye blink recognition method using feature level fusion (M_mERM_FLF) is proposed. A new feature: eye-eyebrow facet ratio (EE_BFR) (formed by fusing eye facet ratio (EFR: ratio of diagonal length and width of eye) and eyebrow to nose facet ratio (E_BNFR: distance between eyebrow landmarks and nose landmark)) for approximating the eye state is computed. Initially, an improved intellectual framework (Sagacious Information Recuperation Technique) that senses the emergency state using information retrieved from eye blinks, pulse rate as well as behavioral patterns(emotions) exhibited by an individual is presented. Further a novel multimodal method (M_mERM_FLF) for detection and counting of eye blinks is implemented. For training, one state-of-the-art database—ZJU is used. To additionally improve the performance, feature-level fusion schemes [simple concatenate and fusion codes (gaborization)] are enforced and equated. Receiver operating characteristics, error rate, sensitivity, specificity, and precision are used to demonstrate the performance of the proposed method qualitatively and quantitatively. Accuracy with proposed M_mERM_FLF is increased to 99.02% (using EE_BGFR method with bagged ensemble classifier) in comparison to unimodal eye blink recognition system (97.60%). 99.80% genuine blinks are classified by M_mERM_FLF (when gaborization fusion is used) using simple tree classifier.

Appendices

Appendix

1.1 A. SIRT

Figure 15 shows the framework for the proposed SIRT (Lamba and Virmani 2018a). The framework unfurls the intact structure of the procedure consolidated together. For instance, if the eye blink sensor detects a deliberate/purposeful bizarre blink pattern, notwithstanding a Fear or Surprise emotion detected by the emotion recognition sensors in the comparative time allotment alongside an irregular pulse rate, the circumstance is a disturbing one. The detailed working of the framework and related parameters can be learned from (Lamba and Virmani 2018a).

Fig. 15

SIRT Framework

B. Feature extraction and classification for unimodal eye blink recognition system

Eye landmarks (Asthana et al. 2014) are detected for each single frame of the video sequence as shown in Fig. 16.

Fig. 16

Open and closed eye with landmarks detected

The EFR between diagonal length and width of the eye is computed (Lamba and Virmani 2018b).

$$ EFRL1 = \frac{{\left| {\left| {P2 - P5} \right|} \right|}}{{2\left| {|P1 - P4} \right||}} $$

(B.1)

$$ EFRL2 = \frac{{\left| {\left| {P3 - P6} \right|} \right|}}{{2\left| {|P1 - P4} \right||}} $$

(B.2)

$$ EFRL = \frac{{\left| {\left| {EFRL1 + EFRL2} \right|} \right|}}{2} $$

(B.3)

$$ EFRR1 = \frac{{\left| {\left| {P8 - P11} \right|} \right|}}{{2\left| {|P7 - P10} \right||}} $$

(B.4)

$$ EFRR2 = \frac{{\left| {\left| {P9 - P12} \right|} \right|}}{{2\left| {|P7 - P10} \right||}} $$

(B.5)

$$ EFRR = \frac{{\left| {\left| {EFRR1 + EFRR2} \right|} \right|}}{2} $$

(B.6)

$$ EFR = \frac{{\left| {\left| {EFRL + EFRR} \right|} \right|}}{2} $$

(B.7)

P1 to P6 are the landmark (left eye) as depicted in Fig. 16, similarly P7 to P12 are the landmark detected in right eye. EFRL and EFRR denotes eye facet ratios for left and right eye respectively. Each individual has a different eye structure; hence the distance is calculated diagonally [diagonal distance between landmarks provide better results (Lamba and Virmani 2018b)]. Since both eyes blink synchronously, the EFR’s of both eyes are averaged (Eq. B.7). To the best of our knowledge, none of the existing paper is taking the diagonal distance between the eye landmarks.

For each video frame, the landmarks are perceived. The EFR is computed between diagonal length and width of the eye. The EFR is primarily constant for an open eye for an individual and approximately drops to zero (less than 0.21) for a closed eye. It is experimentally found that the when an individual eye is closed (during blink), EFR value is less than or equal to 0.21 (threshold value). Figure 17 shows frame wise eye status (open/half open/close) with EFR values of a video sequence (frame 13 to 25) from ZJU database. Blink starts with frame 16 and ends with frame 20. It can be clearly seen from the figure that the moment eye is in half open or open state, EFR value is greater than 0.21. Same pattern is seen for all the videos of the database (ZJU). Another instance of an EFR signal over the video sequence (147 frames) consisting of six blinks (as the EFR wave has dropped six times below the threshold value (0.21)) is presented in Fig. 18.

Fig. 17

Frame wise eye status (open/half open/close) with EFR values

Fig. 18

An instance of detected blink. Input image (ZJU database) with detected landmarks, EFR plot over a video sequence (portrayed frame marked with red line)

Theoretically, to find the landmark (P2, P3, P5 and P6) coordinates, eye structure (eye landmarks) can be linked to an ellipse, translated from the origin by a distance (h, k); the equation of the ellipse is given by

$$ \frac{{\left( {x - h} \right)^{2} }}{{a^{2} }} + \frac{{\left( {y - k} \right)^{2} }}{{b^{2} }} = 1 $$

(B.8)

To find the coordinates (P2, P3, P5 and P6), a line (slope m = 1) is made to pass through center of the ellipse (h, k), which is given by

$$ y = mx + \left( {k - h} \right) $$

(B.9)

By plugging Eq. B.9 in Eq. B.8 we get,

$$ \frac{{\left( {x - h} \right)^{2} }}{{a^{2} }} + \frac{{\left( {x - h} \right)^{2} }}{{b^{2} }} = 1 $$

(B.10)

which may be rearranged as a quadratic equation given by

$$ \left( {a^{2} + b^{2} } \right)x^{2} + \left( { - 2h\left( {a^{2} + b^{2} } \right)} \right)x + h^{2} \left( {b^{2} + a^{2} } \right) - a^{2} b^{2} = 0 $$

(B.11)

Thus the coordinate of intersections is given by

$$ x_{1,2} = \frac{{2h\left( {a^{2} + b^{2} } \right) \pm 2ab\sqrt {a^{2} + b^{2} } }}{{2(a^{2} + b^{2} )}} $$

(B.12)

$$ y_{1,2} = x_{1,2} \pm \left( {k - h} \right) $$

(B.13)

2.1 Classification of feature extracted

If a subject yawns or voluntary closes the eyes for a long period or even makes a facial expression, a series of low EFR values are received. This does not signify that the subject is blinking. Therefore, a classifier that takes a larger temporal window of a frame as an input is proposed. Owing to the fact that a normal blink interval lasts from 100 to 400 ms, roughly 420 ms window can have a momentous impact on a blink detection. Further the database (ZJU) contains videos recorded at 30 fps. Thus, to get the best results a 13-dimensional feature is grouped by concatenating the EFRs of its ± 6 neighboring frames. For training, 13 EFR values (13 consecutive frames) are taken into account and assigned the response value as 1 if a blink is present, otherwise 0. A sample classification matrix for a video sequence with 1000 input frames is shown in Fig. 19. Here EFRi (i = 1 to number of frames) is the EFR computed for the ith frame of the video sequence. For example, if there are 1000 frames in a video sequence, the first row will contain EFR values for first 13 frames (1–13), second row will contain 13 EFR values (2- 14 frames), and so on. The last row will be having EFR values for frame number 988 to 1000. Different classifiers are trained from the video sequences (ZJU) and manually annotated sequences. While testing, a classifier is executed in a scanning-window fashion. A 13-dimensional feature is computed and characterized by classifiers for each frame of a video sequence. Blink is detected instantly after the blink’s end. Stage wise process for blink detection is shown in Fig. 20.

Fig. 19

Sample training matrix for a video sequence with 1000 frame

Fig. 20

Stepwise process for blink detection

C. Percentage per true class

Tables 6, 7 and 8 shows a particular instance of percentages per true class including true positive rate (TPR) and false negative rate (FNR) (Confusion Matrix) of individual classifiers for EFR (unimodal), EE_BFR and EE_BGFR (multimodal) methods.

Table 7 Confusion matrix for EE_BFR

Table 8 Confusion matrix for EE_BGFR