Pupil trajectory tracing from video-oculography with a new definition of pupil location

Highlights

• A new definition of pupil trajectory tracking in video-oculography is proposed.

• We proposed to use the lower pole of pupil as the location point.

• We designed an adaptive threshold method to segment the pupil.

• We manually annotated the actual BPPV inspection data set as experimental data.

• The proposed method shows good locating accuracy compared to its peer methodologies.

Abstract

Pupil localization is the critical technology of eye tracking, which plays a significant role in many eye movement analysis applications, such as pupil trajectory tracing from video-oculography(VOG). However, when the pupil is occluded, the existing representative pupil center based localization methods are imprecise or even unable to find the pupil. Unfortunately, in actual VOG, the pupil is inevitably covered by the upper eyelid, resulting in a poor effect of these approaches in practice. In order to alleviate this drawback, we initiate a new way to determine the pupil position by locating the lower pole of pupil (LLPP), which can accurately locate the pupil with about 5% of the exposed area. As a result, the proposed method can reduce locating mistakes, increase data utilization, and maintain pupil trajectories continuity. The experimental results in real VOG of the benign paroxysmal positional vertigo (BPPV) patients demonstrate the effectiveness and superiority of the proposed LLPP.

Introduction

In the process of human perception and cognition of the outside world and even psychological activities, human eye movement can reflect rich information. People’s emotion can be observed from eye movement [1], researchers in user study can observe what this person pays attention to [2], and otolaryngologists in hospital can check what kind of vestibular diseases this person would suffer from [3], [4].

With the development of computer vision and artificial intelligence, a subject of vision-based eye movement analysis has emerged and triggered many application scenarios, mainly in the field of consumption and medicine. For instance, using Tobii’s eye tracker, the details of consumers’ attention to goods can be recorded, and the analysis of these data can provide support for providing better products to the market [5]. In VR experience with head mounted displays like HTC Vive and Pico, players can enjoy better feelings benefited from the foveated rendering technique based on eye tracking [6], [7].

During eye examination, doctors often need to examine the reaction of patients’ eyes for diagnosis and treatment, such as stimulating human eyes with flashing light and analyzing the oscillation power of pupil to diagnose glaucoma [8], judging the type of benign paroxysmal positional vertigo (BPPV) by Dix–Hallpike tests [9], measuring gaze fixation and pupil reaction to check Parkinson’s disease [10], and stimulating human eyes with two bicolored objects to test whether patients had Alzheimer’s disease [11].

To analysis eye motion, it is first to locate and track the eye. Pupil is in general selected as a representative to the eye, for pupil is an organ in eye ball of a round shape and a dark color in contrast to its outside organs. And in most cases, pupil center is further taken as the position of the pupil. In the above applications, it is then extremely critical to achieve precise location of the pupil. Existing pupil locating algorithms can be generally divided into three types: feature-based, model-based, and deep learning approaches.

Feature-based approaches use the eye features for location, generally based on the prior knowledge that the pupil is a black ellipse and would change size when stimulated. Timm and Barth [12] considered the intersection point of iris contour’s gradient vectors as the candidate of pupil center, and calculate the most likely point. Xiong et al. [13] estimated the iris center point with eye contour. Model-based approaches are to find the best model by function fitting, such as the ellipse fitting (EF) and model parameter optimization [14], [15], the circular Hough transform (CHT) [16], and vector field convolution model [17]. While these two types of methods are easy to calculate and convenient to use, the locating accuracies are limited. In addition, the parameters in these algorithms need to be manually set according to actual data and rely on the threshold selection of binarization [18].

In recent years, deep learning methods for pupil locating have emerged, which show better robustness than the above methods. Biswas et al. [19] added images variants such as exposure, rotation and defocus blur to the model training data to optimize the model. Lee et al. [20] introduced the attention mechanism into the model to improve the speed of face detection and find the pupil center more quickly. Eivazi et al. [21] designed a model that can locate the pupil center of disturbed images in real time. Yiu et al. [18] proposed a method to estimate the elliptical contour of the pupil and analyze the blink according to the confidence of pupil ellipses. However, these deep learning method needs a large amount of computation and data to train the network parameters [22].

Under normal circumstances, the behavior of human eye movement mainly includes gaze, saccade and smooth pursuit. All these normal movements are actively driven by human vision system. If there are abnormalities, such as BPPV caused by otolith dislocation in patients leading to vertigo, nystagmus will occur. Unlike the normal eye movements, nystagmus is an involuntary behavior of patients. Elder patients are prone to suffer from BPPV [23]. Clinically, doctors determine what type of BPPV the patient has by observing the patient’s nystagmus. Doctors can be assisted with a set of eye tracking system [24] in examining patients’ nystagmus. With a head mounted video-oculography (VOG) device, the raw data of nystagmus can be recorded by the infrared camera embedded therein [25].

During the examination, in order to induce expected nystagmus, doctors need to manually adjust patient’s posture, which would cause the head mounted device shift [26]. Once the pupil exceeds the boundary of the image, the related frames are of no usage. Patients usually blink frequently and have difficulty opening their eyes completely due to their symptoms. This may cause the pupil severely occluded. Moreover, the brightness of the infrared light has to be readjusted to start a new examination, which causes the recorded VOG of different illumination.

Blinking, as a physiological phenomenon, occurs frequently. During the eye blinking, the pupil is usually occluded so that the pupil contour becomes irregular. This causes the features available for further analysis insufficient. Therefore, the accuracy of pupil position calculated by feature-based and model-based methods will be reduced. For the deep learning methods, the remaining part of the pupil occluded is treated as an integral but estimated pupil, so that the resulted parameters cannot be satisfactory. The larger the occluded part of the pupil, the less accurate the estimated pupil center is. If the occluded part of the pupil reaches some extent, the corresponding frame would be dropped for it becomes useless in eye movement analysis.

In the final analysis, these deficiencies are caused by using the center point to represent the pupil. To alleviate all these drawbacks, in this paper, we introduce the lower pole of the pupil as its representative point instead of pupil center. With a set of delicate tools in image processing, an effective feature-based algorithm for locating the lower pole of pupil(LLPP) is designed. In order to test the functionality of this algorithm, we create an infrared eye movement data set from videos of patients captured during BPPV examinations. The trajectory based on the lower poles of the pupil can express the eye movement more precisely and make more frames take effect. Training the BPPV classification model with pupil trajectory generated by LLPP can make the classification model learn more practical features and obtain higher accuracy.

The rest of this paper is organized as follows. We first describe the LLPP algorithm in detail in Section 2, then carry out experiments in Section 3 to verifies the effect of the LLPP algorithm, and finally in Section 4 conclude this paper.