Pupil trajectory tracing from video-oculography with a new definition of pupil location
Introduction
In the process of human perception and cognition of the outside world and even psychological activities, human eye movement can reflect rich information. People’s emotion can be observed from eye movement [1], researchers in user study can observe what this person pays attention to [2], and otolaryngologists in hospital can check what kind of vestibular diseases this person would suffer from [3], [4].
With the development of computer vision and artificial intelligence, a subject of vision-based eye movement analysis has emerged and triggered many application scenarios, mainly in the field of consumption and medicine. For instance, using Tobii’s eye tracker, the details of consumers’ attention to goods can be recorded, and the analysis of these data can provide support for providing better products to the market [5]. In VR experience with head mounted displays like HTC Vive and Pico, players can enjoy better feelings benefited from the foveated rendering technique based on eye tracking [6], [7].
During eye examination, doctors often need to examine the reaction of patients’ eyes for diagnosis and treatment, such as stimulating human eyes with flashing light and analyzing the oscillation power of pupil to diagnose glaucoma [8], judging the type of benign paroxysmal positional vertigo (BPPV) by Dix–Hallpike tests [9], measuring gaze fixation and pupil reaction to check Parkinson’s disease [10], and stimulating human eyes with two bicolored objects to test whether patients had Alzheimer’s disease [11].
To analysis eye motion, it is first to locate and track the eye. Pupil is in general selected as a representative to the eye, for pupil is an organ in eye ball of a round shape and a dark color in contrast to its outside organs. And in most cases, pupil center is further taken as the position of the pupil. In the above applications, it is then extremely critical to achieve precise location of the pupil. Existing pupil locating algorithms can be generally divided into three types: feature-based, model-based, and deep learning approaches.
Feature-based approaches use the eye features for location, generally based on the prior knowledge that the pupil is a black ellipse and would change size when stimulated. Timm and Barth [12] considered the intersection point of iris contour’s gradient vectors as the candidate of pupil center, and calculate the most likely point. Xiong et al. [13] estimated the iris center point with eye contour. Model-based approaches are to find the best model by function fitting, such as the ellipse fitting (EF) and model parameter optimization [14], [15], the circular Hough transform (CHT) [16], and vector field convolution model [17]. While these two types of methods are easy to calculate and convenient to use, the locating accuracies are limited. In addition, the parameters in these algorithms need to be manually set according to actual data and rely on the threshold selection of binarization [18].
In recent years, deep learning methods for pupil locating have emerged, which show better robustness than the above methods. Biswas et al. [19] added images variants such as exposure, rotation and defocus blur to the model training data to optimize the model. Lee et al. [20] introduced the attention mechanism into the model to improve the speed of face detection and find the pupil center more quickly. Eivazi et al. [21] designed a model that can locate the pupil center of disturbed images in real time. Yiu et al. [18] proposed a method to estimate the elliptical contour of the pupil and analyze the blink according to the confidence of pupil ellipses. However, these deep learning method needs a large amount of computation and data to train the network parameters [22].
Under normal circumstances, the behavior of human eye movement mainly includes gaze, saccade and smooth pursuit. All these normal movements are actively driven by human vision system. If there are abnormalities, such as BPPV caused by otolith dislocation in patients leading to vertigo, nystagmus will occur. Unlike the normal eye movements, nystagmus is an involuntary behavior of patients. Elder patients are prone to suffer from BPPV [23]. Clinically, doctors determine what type of BPPV the patient has by observing the patient’s nystagmus. Doctors can be assisted with a set of eye tracking system [24] in examining patients’ nystagmus. With a head mounted video-oculography (VOG) device, the raw data of nystagmus can be recorded by the infrared camera embedded therein [25].
During the examination, in order to induce expected nystagmus, doctors need to manually adjust patient’s posture, which would cause the head mounted device shift [26]. Once the pupil exceeds the boundary of the image, the related frames are of no usage. Patients usually blink frequently and have difficulty opening their eyes completely due to their symptoms. This may cause the pupil severely occluded. Moreover, the brightness of the infrared light has to be readjusted to start a new examination, which causes the recorded VOG of different illumination.
Blinking, as a physiological phenomenon, occurs frequently. During the eye blinking, the pupil is usually occluded so that the pupil contour becomes irregular. This causes the features available for further analysis insufficient. Therefore, the accuracy of pupil position calculated by feature-based and model-based methods will be reduced. For the deep learning methods, the remaining part of the pupil occluded is treated as an integral but estimated pupil, so that the resulted parameters cannot be satisfactory. The larger the occluded part of the pupil, the less accurate the estimated pupil center is. If the occluded part of the pupil reaches some extent, the corresponding frame would be dropped for it becomes useless in eye movement analysis.
In the final analysis, these deficiencies are caused by using the center point to represent the pupil. To alleviate all these drawbacks, in this paper, we introduce the lower pole of the pupil as its representative point instead of pupil center. With a set of delicate tools in image processing, an effective feature-based algorithm for locating the lower pole of pupil(LLPP) is designed. In order to test the functionality of this algorithm, we create an infrared eye movement data set from videos of patients captured during BPPV examinations. The trajectory based on the lower poles of the pupil can express the eye movement more precisely and make more frames take effect. Training the BPPV classification model with pupil trajectory generated by LLPP can make the classification model learn more practical features and obtain higher accuracy.
The rest of this paper is organized as follows. We first describe the LLPP algorithm in detail in Section 2, then carry out experiments in Section 3 to verifies the effect of the LLPP algorithm, and finally in Section 4 conclude this paper.
Section snippets
A new approach to locate the pupil
In this section, we present the detail of the LLPP approach. Consider a video clip captured during the examination of a BPPV patient as of frames, each of width and height . Our goal is to find the lower pole of the pupil in each frame ; the sequence of all the lower poles then forms a trajectory of eye movement.
As shown in Fig. 1, to complete this task, we conduct the following steps: conduct image preprocessing to improve the quality of
Experimental analysis
To demonstrate the validity of the proposed LLPP algorithm, we compare it with other representative center based location algorithms on the real nystagmus video dataset. The methods used for comparison include the ellipse fitting (EF) [14], the circular Hough transform (CHT) [16] and the deep learning method in [21] (DL for short).
The EF method uses the least square criterion to obtain the ellipse that is most suitable for the points on the pupil contour. The CHT method uses pupil contour to
Conclusion
In this paper, an effective pupil location method for pupil trajectory tracing from Video-Oculography (VOG) was proposed, named LLPP. Different from most of the pupil center location methods, LLPP locates the lower pole of pupil instead of locating the pupil center. The proposed method was evaluated with other pupil center locating methods on the real VOG data. The experimental results validated that the proposed method had higher utilization of data, could maintain the consistency of locating
CRediT authorship contribution statement
Weike Deng: Investigation, Conceptualization, Methodology, Software, Visualization, Writing – original draft. Jingru Huang: Validation, Data curation. Sheng Kong: Software, Data curation. Yinwei Zhan: Supervision, Formal analysis, Writing – review & editing. Jujian Lv: Formal analysis, Writing – review & editing. Yong Cui: Validation, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by Natural Science Foundation of Guangdong Province, China [grant number 2019B010150002] and [grant number 2020B0101130019].
References (31)
SPIDERS plus : A light-weight, wireless, and low-cost glasses-based wearable platform for emotion sensing and bio-signal acquisition
Pervasive Mob. Comput.
(2021)Foveated AR: dynamically-foveated augmented reality display
ACM Trans. Graph.
(2019)DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning
J. Neurosci. Methods
(2019)Deep learning based torsional nystagmus detection for dizziness and vertigo diagnosis
Biomed. Signal Process. Control
(2021)- et al.
Micropursuit and the control of attention and eye movements in dynamic environments
J. Vis.
(2021) - et al.
Towards providing an automated approach to differentiating the nystagmus of meniere’s disease, vestibular migraine, and benign paroxysmal positional vertigo
Otol. Neurotol.
(2021) Feasibility of eye tracking assisted vestibular rehabilitation strategy using immersive virtual reality
Clin. Exp. Otorhinolaryngol.
(2019)- et al.
The tobii pro spectrum: A useful tool for studying microsaccades?
Behav. Res. Methods
(2021) - et al.
Eyevr: low-cost VR eye-based interaction
Gaze-contingent flicker pupil perimetry detects scotomas in patients with cerebral visual impairments or glaucoma
Front. Neurol.
(2018)
Detection of saccades and quick-phases in eye movement recordings with nystagmus
Fixation duration and pupil size as diagnostic tools in Parkinson’s disease
J. Parkinson’s Disease
Oculomotor behaviors and integrative memory functions in the alzheimer’s clinical syndrome
J. Alzheimers Disease
Accurate eye centre localisation by means of gradients
Remote gaze estimation based on 3D face structure and iris centers under natural light
Multimedia Tools Appl.
Cited by (2)
Reliable Eye-Blinking Detection With Millimeter-Wave Radar Glasses
2024, IEEE Transactions on Microwave Theory and TechniquesRadar Assistive System for People with Neurodegenerative Disorders Through Head Motion and Eyes Blinking Detection
2023, IEEE MTT-S International Microwave Symposium Digest
- 1
Contributed equally to the work.