Deep-learning-based reading eye-movement analysis for aiding biometric recognition
Introduction
Biometric recognition technology is widely used in e-commerce, electronic products, and network security. Reading eye-movement is a human behavior with the biometric characteristics of convenience, security, universality and collectability [1]. Thus, users can be identified by comparing human eye-movement trajectories.
Based on measurements of eye-movement trajectory similarity, the existing eye-movement recognition technology extracts the measurable features of the eye-movement trajectory, including the fixation duration and the lengths of the saccades. Other technologies compare the eye-movement trajectories using more complicated space and time information of the eye-movement. The existing technologies are usually used to obtain the similarity measurement value (or values) of eye-movement trajectories. However, the characteristics of the stimuli are not considered.
Related studies on reading psychology have shown that human eye-movements during reading are significantly different between individuals [2], [3], [4], but the same individual exhibits a certain similarity. This suggests that human eye-movement is unique and stable to some extent and can be used in biometrics [5], [6]. Fig. 1 shows the fixation sequences as ten subjects read the same text.
In this study, multiple-input deep neural networks were utilized to learn the reading eye-movement behaviors and construct a computational model for reading eye-movement recognition (REMR). The model can learn the features of the stimuli (reading materials) and the eye-movement trajectory. The model can fully simulate human eye-movement after training and can be applied for user identification by comparing the predicted and actual (to be identified) fixation sequences. Combined with other biometric recognition technologies, this technology can be used as a supplementary tool for the existing identity authentication methods to realize multi-factor identity authentication.
The main contributions of the present paper are as follows.
- 1.
A biometric recognition technology is proposed based on reading eye-movement. Accounting for the stimuli (reading materials) and scanning path, this technology uses fewer handcrafted features to obtain effective recognition by utilizing the deep-learning characteristics of automatic feature extraction. As a result, the model obtained an 86.5% Rank-1 recognition accuracy on the test set.
- 2.
A REMR computational model based on deep-learning is proposed. This model uses a deep neutral network to generate the predicted fixation sequence and measure the similarity distance between the predicted and actual fixation sequences to identify the subject.
- 3.
An algorithm is presented for evaluating the fixation sequence similarity. The algorithm uses dynamic time warping (DTW) to measure the similarity between two fixation sequences, and the proposed algorithm obtained an equal error rate (EER) of 19.4% on the test set.
Section snippets
Related work
The biometric study of eye-movement stems from the early study of scanpath theory, in which the word “scanpath” refers to the space path formed by an orderly fixation and saccadic sequence. In 1971, Noton and Stark [7] found that the general scanpath followed by a subject during the first viewing of a pattern was repeated in the initial eye-movements of roughly 65% of subsequent viewings, and the scanpath for specific stimuli varied from person to person.
The 2004 paper by Kasprowski and Ober [8]
Problem setting
Experimental results of eye-movement and reading have indicated that eye-movement during reading is goal-oriented and discrete [4]. This means that the saccade is non-random in selecting a visual target and that the saccade target points to a particular word rather than a specific distance. Based on this notion, there are many candidate words in the latent saccade period, and each word has a chance to be selected as the target of subsequent saccades. In this work, the probability of each word
Experimental environment and dataset
The experimental environment was Python3.7 + Keras2.2.4 + TensorFlow1.13. All the code was released on GitHub at https://github.com/wxmgo/eye_movement_in_reading/. In addition to being interesting to readers in the fields of biometrics and machine learning, the public posting of the code on Github will allow others to easily utilize or modify this method for their own problems.
We obtained the dataset from the Provo Corpus [34]. The corpus is open and can be downloaded from the Open Science
Evaluation metric
The proposed method was evaluated by the Rank-1 (R1) accuracy rate and the equal error rate (EER). The Rank-1 accuracy rate is the ratio of the total number of correct recognitions to the number of samples. The EER is the value when the false acceptance rate (FAR) and the false rejection rate (FRR) are equal.
The false acceptance rate is the ratio that is considered to be the same subject when the similarity distance of different eye-movement sequences is greater than the given threshold during
Conclusion
In this paper, a type of reading eye-movement biometric recognition technology was proposed based on deep-learning. This technology constructs a reading eye-movement recognition (REMR) computational model based on a multi-input deep neural network and identifies human subjects by comparing the predicted and actual fixation sequences. This model is less dependent on the data features and requires less pre-possessing, which makes it attractive for industrial and engineering applications [42]. The
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China [grant number 61871326], the Humanities and Social Science Fund of Ministry of Education of China [grant number 18YJCZH180] , the Natural Science Foundation of Shaanxi Province [grant number 2018JM6116], the Social Science Foundation of Shaanxi Province [grant number 2019M001] , and the Aeronautical Science Foundation of China [grant number 20185153].
Xiaoming Wang received the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China, in 2020. He is currently an Associate Professor with Xi'an International Studies University. His research interests include cognitive computing and artificial intelligence.
References (42)
- et al.
Eye movements during multiple object tracking: where do participants look?
Cognition
(2008) - et al.
Detecting cyberattacks in industrial control systems using online learning algorithms
Neurocomputing
(2019) - et al.
Eye-movement strategies in developmental prosopagnosia and “super” face recognition
Q. J. Exp. Psychol.
(2017) - et al.
Saccade target selection in Chinese reading
Psychon. Bull. Rev.
(2015) Eye movements in reading and information processing: 20 years of research
Psychol. Bull.
(1998)Book review: eye tracking: a comprehensive guide to methods and measures
Q. J. Exp. Psychol.
(2016)- et al.
Eye movements in reading and information processing: Keith Rayner’s 40 year legacy
J. Memory Lang.
(2016) - et al.
Limits on lexical prediction during reading
Cogn. Psychol.
(2016) - et al.
Scanpaths in eye movements during pattern perception
Science (80-)
(1971) - et al.
Eye movements in biometrics
Lect. Notes Comput. Sci. (Including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics)
(2004)
Taming the electronic tiger: report on the 2004 ala midwinter symposium on electronic resource management
Ser. Rev.
Biometric identification based on the eye movements and graph matching techniques
Pattern Recognit. Lett.
GANT: Gaze analysis technique for human identification
Pattern Recognit.
Computational models of reading: a primer
Lang. Linguist. Compass.
The E-Z reader model of eye-movement control in reading: comparisons to other models
Behav. Brain Sci.
Swift: a dynamical model of saccade generation during reading
Psychol. Rev.
Cited by (10)
Pupil localization algorithm based on lightweight convolutional neural network
2024, Visual ComputerApplication of AI Technology in Online Platforms Based on Cognitive Emotion Regulation
2023, Frontiers in Artificial Intelligence and ApplicationsBiometric Recognition Based on Recurrence Plot and InceptionV3 Model Using Eye Movements
2023, IEEE Journal of Biomedical and Health InformaticsEye movement pattern of reading Chinese Classics
2023, Digital Scholarship in the Humanities
Xiaoming Wang received the Ph.D. degree from Northwestern Polytechnical University, Xi'an, China, in 2020. He is currently an Associate Professor with Xi'an International Studies University. His research interests include cognitive computing and artificial intelligence.
Xinbo Zhao received the Ph.D. degree from the Northwestern Polytechnical University, Xi’an, China, in 2003. Currently, he is a professor at the School of Computer Science, Northwestern Polytechnical University. His interests include image processing, computer vision, pattern recognition and artificial intelligence. He is the author or co-author of more than 100 scientific papers.
Yanning Zhang received her B.S. degree from Dalian University of Science and Engineering in 1988, M.S. and Ph.D. Degree from Northwestern Polytechnical University in 1993 and 1996 respectively. She is presently a Professor of School of Computer Science and Technology, Northwestern Polytechnical University. She is also the organization chair of ACCV2009 and the publicity chair of ICME2012. Her research work focuses on signal and image processing, computer vision and pattern recognition. She has published over 200 papers in these fields, including the ICCV2011 best student paper. She is a member of IEEE.