VPID: Towards Vein Pattern Identification Using Thermal Imaging

2.1 Biometric Authentication

The uniqueness of biometrical features has been widely explored to identify and authenticate users [9], [23], [32]. Biometrics are classified into behavioral and physiological biometrics [41]. Behavioral biometrics rely on behavioral cues to authenticate users, for instance, their touch and keystroke behavior [13], [16], [33]. Physiological biometrics are based on “something you are” and include a person’s physiological information such as iris, face, voice, fingerprint, and hand geometry [37]. An example is Bodyprint [19] that uses body parts like ear, finger, fist, and palm prints, obtained via a mobile phone touchscreen as a biometric trait.

Veins patterns are another biometric feature that has been proposed for identification and authentication. Since vein patterns are unique for each users and stable for a lifetime [1], they are well suited as a biometric. For example, veins can be captured under IR-lights [5], using VGA [39], or near IR imaging [17]. While these technologies provide good input, they either require external sources of illumination [38], have high costs, or require multiple sensors [14].

Researchers also explored using thermal (far IR) imagining to extract the veins on the palm [27], [40]. As veins have different temperature than the surrounding skin [11], they are visible to thermal cameras without additional illumination. Vein triangulation and knuckle shapes are used to differentiate between users. From this we learn that veins are a powerful means for authentication. At the same time, approaches are required that allow authentication using vein patterns to be seamlessly integrated with interactive systems.

2.2 Thermal Imaging for Sensing Interaction

Due to the advantages that thermal imaging provides, including robustness to illumination and color changes, it has recently been proposed as a sensing technology for interactive systems [3], [29], [34]. This is achieved by integrating thermal imaging and existing computer vision techniques to improve touch or gestural interaction. The advantages of thermal imaging are utilized to overcome common RGB and depth cameras’ limitations, such as light dependence and lack of visible differences between veins and skin. Larson et al. used heat traces caused by fingers touching a surface and detected pressure for interaction on surfaces [29]. Sahami Shirazi et al. proposed thermal reflection to expand the interaction space beyond the camera’s direct field-of-view [34]. Abdelrahman et al. investigated the material properties supporting both touch and mid-air gestural interaction using thermal imaging as sensing technology [3]. Furthermore, Woo Cho et al. [7] introduced a new system to detect the thermal handprint for forensic identification.

Thermal imagers have been used in security contexts before. Abdelrahman et al. presented an approach to reveal PIN and lock patterns from thermal images of a user’s mobile device [2]. Overall, thermal cameras are promising sensors for a range of interactive systems e. g., multiple user interaction, tabletop interactions [34] that would naturally benefit from seamless identification and authentication.

3.1 Vein Pattern Construction Pipeline

For extracting the hand veins, we are using the OpenCV library^[2] for image processing and features extraction. We apply in total nine steps to a thermal image stream to construct the vein pattern of each user that we later use to authenticate a user.

1. Image extraction: First, we extract frames of raw data from the live-stream of the thermal camera (cf., Figure 2a).

2. False-color image transformation: We transform the acquired raw thermal data, represented by an array of unsigned short integers, into a false-color image using an iron color palette scaled from minimumtemperature −7 and maximumtemperature +1. This palette boosts the color difference of temperatures across the shading palette without enhancing or omitting any potential temperature [18] (cf., Figure 2b). This insures that all small details are captured. Afterwards, we apply a color-based threshold to remove any background noise from the hand by setting it to black (cf., Figure 2c).

Figure 2

The first half of the steps in our pipeline that would result in the region of interest.

Figure 3

Starting with the region of interest, the second half of the image processing pipeline is presented. Specially, the area from the region of interest is processed into a skeleton.

3. Hand-detection: Then, we use the data from step 2 to detect the warmest area in the image. A convex hull is then drawn to connect all the local maxima of the warm points. Next, we detect the hand informalities (i. e., fingers and knuckles) to verify a hand shape. We measure the defects, which are the points with the largest distance from the convex set (cf., Figure 2d). If no defects are detected, the shape is not identified as a hand.

4. Region of interest (ROI): As the vein pattern is located on the back of the hand, we define the ROI as the circle with the largest diameter enclosed in the hand’s contour (cf., Figure 2e). For that, we try all possible combinations of circles across the hand area (i. e., warmest area). To reduce random noise and artifacts at the border of the circle, the diameter is hence multiplied with a constant factor of 0.95 and we extract that mask (cf., Figure 2f).

5. Extraction of the vein pattern skeleton: We transfer the circular mask depicting the ROI to grayscale image with a normalized histogram. Then, we use an adaptive-threshold based approach to detect the local maxima and assigning them to be part of the vein skeleton (cf., Figure 3a).

6. Noise removal: Afterwards, we use median blurs followed by an opening morphological operation to eliminate noise and small, single artifacts from the image (cf., Figure 3b).

7. Topological skeleton: The image obtained from step 6 results in thick lines or broad areas (i. e., one or multiple connected lines with a width of more than one pixel). Hence, we used the thinning approach of Zhang Suen [42], which reduces the size of each area until it is exactly one pixel thin (cf., Figure 3c).

8. Skeleton affine transformations: Using the center of the ROI obtained in step 4, we obtain the center point of the defects in step 3 and plot a vector. This vector is used to rotate any captured image upwards, in the direction of a positive y-axis (cf., Figure 3d). To further unify the metrics of the logged data, we scale the ROI so that the diameter reaches the maximum height of the image (cf., Figure 3e).

9. Skeletons stacking: In this final step, the vein skeletons obtained from several images of the same video sequence (i. e., same hand) are stacked over each other to assure the best quality to the finest detail. We refined the first 14 frames of each video and stacked them as a final step.

3.2 Vein Pattern Authentication

The authentication process is done through comparing the similarities between vein pattern skeletons. We implemented six different approaches for shape comparison, namely, Hausdorff distance, Shape Context, Jaccard distance, and three different versions of Hu moments. In addition, we refine our results by machine learning algorithms.

Hausdorff and Shape Context distances based on the work of Huttenlocher, Klanderman and Rucklidge [21], and Belongie et al. [6], respectively. These two approaches depend mainly on comparing the distance-difference between two skeletons. Hence, a user is authenticated if the comparison of two skeletons, S1 and S2 results in distance(S1,S2)≤Vthreshold.

Jaccard distance . The Jaccard index [22] defines a metric for similarity between two sets (e. g., A and B). Where A consists of all white pixels of the first skeleton, and B consists of all white pixels of the second skeleton. Each pixel is defined as a tuple of its X- and Y-coordinates. The Jaccard index is then defined as: J(A,B)=∣A∩B∣∣A∪B∣, where 0≤J(A,B)≤1. In other words, the size of the overlapping area between two images is divided by the area of the union of both images, where the area is defined by the presence of the white pixels in the binary image. To provide a comparable metric the Jaccard distance is used. It is defined as: JDistance(A,B)=1−J(A,B).

Hu moments based on the work of Hu [20]. These are numerical values retrieved from image characteristics (e. g., lie on the same plane, has non-zero values, etc.). These numerical values then can be compared to each other. Hu proposed three different ways, namely:

1. I1(A,B)=∑i=1...7|1miA−1miB|

2. I2(A,B)=∑i=1...7|miA−miB|

3. I3(A,B)=maxi=1...7|miA−miB||miA|

Convolutional Neural Network . Recent work using deep learning has shown that Convolutional Neural Network (CNN) can be used to classify images [25], [30]. A trained CNN can match one new, unknown image onto one of many learned classes of images. When used as standalone mechanism, the input data would always be identified and classified to any of the registered classes. Therefore, we deployed the CNN with the other comparison and matching algorithms. In our case, this can be used to train the system with the skeleton images of the user’s vein pattern. For our work, we tested various CNNs (i. e., LeNet [31], AlexNet [26], and VGG-16 [36]) with sample data collected across several pilot studies. Finally, we decided to use “AlexNet” for our evaluation, as it does not require long training time and has high accuracy. The network was trained for 30 epochs, with a constant random seed, stochastic gradient descent as the solver and a base learning rate of 0.001.

4.1 Participants and Procedures

We recruited 12 participants (4 female, 8 male, μ=28, σ=8.5) via university mailing lists. The participants have backgrounds in different majors. After welcoming the participants, we described the goal of the study and handed out consent forms as well as demographic questionnaires. We asked the participant before recording to wash his/her hands under normal tab water. This procedure was done at the beginning to unify the temperature of participant’s hands. The study was conducted over two sessions on the same day, separated by 2–3mins break. In every session, each hand (i. e., left or right) was recorded for 20secs. We defined the area and direction in which the hand should be placed on and pointing to, so the camera could start recording.

Collecting biometric information from participants raised the need for a secure procedure, in order to protect their privacy. As the always existing, possible risk of a data breach leaves participants with no way of changing or revoking their biometric information, a failure to comply with a secure procedure to protect their data is highly unethical. Subsequently, we gathered the informed consent of all our participants and assigned them an anonymous identifier. Directly upon finishing the acquisition of data per participant, we deleted the connecting attribute of their private information to the anonymous identifier, so that no connection can be re-established in the future.

4.2 Evaluation

Figure 5

Examples of the six image-quality rating criteria mentioned in Table 1.

To start with the evaluation process, we had to first categorise the received images. One major aspect that directly influences the obtained results is the quality of the images captured by the thermal camera. In general, we noticed that certain circumstances exist that result in a “good” hand thermal-image recording, which contain a very visible vein pattern. In contrast to other circumstances, a “bad” thermal recording with the vein pattern being only barely visible. The quality of a captured image is defined by the visibility of the vein pattern in the thermal image, which can be measured by its contrast towards the surrounding area of tissue. One main contributing factor is the angle of the hand towards the thermal imager. If the hand is twisted or curled while being captured by the thermal imager, the vein pattern might not be visible at all. One of the core assumptions of our vein-detection algorithm is that the vein pattern would be in general warmer (i. e., brighter) than the surrounding area. Hence, a quantifying quality metric had to be set to categorize any input. For that we used the manual Thermal Imagining Quality Rating (TIQR) algorithm, where we defined six main criteria. These are based on observations from several pilot studies, where we captured one or two hands and extracted the skeletons. Then we matched visually what we obtained compared to the original input. Each criterion is assigned a numerical value. Thus, the TIQR value would be the summation of all the existing criteria ratings. Also, if the final TIQR value was found to be negative, it would be rounded up to 0.

By doing that, we ended up with four main categories of images, where only the categories having a TIQR of “3” or “4” is furthermore processed for vein pattern detection. All the data, that was marked for processing was subsequently classified into one of three groups. The first group is the “registered” data set. The data in this set was provided to the system and used to train an “AlexNet”. It consists of, in total, twelve distinct skeletons, where each skeleton originates from a different person’s hand. The second group is the “testing” data set. It consists of six elements, where each one has a counterpart in the “registered” data set. These are the skeletons extracted from the second recording session of the same registered hand. The third group is formed only of the skeletons of not-registered hands and is used also for testing, referred to as the “unknown-testing” data set.

Figure 6

An activity chart depicting all three variants of the authentication system.

For the evaluation, the “testing”- and “unknown-testing”-skeletons are tested against the authentication system. Dependent on the input and the system’s output, the true positive rate (“TPR”), true negative rate (“TNR”), false positive rate (“FPR”) and false negative rate (“FNR”) are determined. Ideally, all skeletons in the “unknown-testing” data set should be rejected by the system and become a true negative (“TN”), as they have no counterpart in the systems’ “registered” data set. A false negative (“FN”) is met, when an element of the “testing” data set is incorrectly rejected, since each element should have been accepted instead. A true positive (“TP”) is counted, if an element of the “testing” data set is correctly accepted by the system. Last but not least, a false positive (“FP”) is met, when an element of the “unknown-testing” data set is accepted by the system. As this data set does not have any counterparts in the “registered” data set, all of them should be rejected instead.

Figure 6 depicts the three evaluated variants of how the authentication system can be designed. The first variant (i. e., Variant I), depicted in Figure 6a, only uses a shape comparison algorithm as a measure of similarity between an input skeleton and any of its registered skeletons. The user’s identity is assumed as the registered skeleton with the closest distance to the input. If the measure of similarity, the “distance”, is below the threshold T1, the authentication attempt is successful, otherwise it is rejected.

The second variant (i. e., Variant II), which is illustrated in Figure 6b, additionally performs an image classification to detect the user’s identity. For the classification, the pre-trained AlexNet is utilized, which states a top prediction and an associated certainty. If this certainty is below T2, the user is rejected. Otherwise, the user’s identity is assumed to be the prediction of the neural network. The threshold T2 is set to a constant value of 75 %.

The third variant (i. e., Variant III), switches the process and exchanges the activities of image classification and shape comparison and is depicted in Figure 6c. Here, the neural network filters the registered data for a set of potential matches. Then, the shape comparison is applied to compare the input against the best matching image of this set only.

As a measure of comparison between the different approaches to utilize the system, we chose the accuracy (“ACC”, see Equation 1a). To find the best suitable values for the threshold T1, we used an optimizer, which calculates the accuracy for every present shape comparison value. The threshold value, which lead to the highest accuracy per method, was then set as the constant value of T1.

4.3 Results

Figure 7

Depiction of the maximum accuracy per variant, when skeleton stacking is not applied.

Figure 8

Depiction of the maximum accuracy per variant, when skeleton stacking is applied.

All three variants, in which the system can be deployed, are depicted in Figure 7, when skeleton stacking is not applied. The bar chart reports the accuracy per shape comparison algorithm. In contrast, Figure 8 depicts the variants, when skeleton stacking is applied.

Figure 9

The FAR/FNR-chart for the authentication system using Variant III, skeleton stacking and the Hausdorff distance as a shape comparison algorithm.

Skeleton stacking improved the performance, when the Hausdorff distance was used as a shape comparison algorithm. Here, combining skeleton stacking and Variant III. leads to an accuracy of 100 %, i. e. the system did not make any errors, when the threshold T1 is set to a value between 9.48 to 10.63. In any other case, skeleton stacking did not improve the accuracy. When no skeleton stacking is applied, the shape comparison methods “Shape Context” and “Jaccard” worked best, yielding the highest accuracies in direct comparison to the Hu-methods and the Hausdorff distance. The Variant III, with the application of skeleton stacking and the Hausdorff distance as a shape comparison algorithm, was able to outperform any other combination, by reaching an equal error rate (“EER”) of zero. Out of the six skeletons in the “testing” data set, all were accepted, while all six skeletons in the “unknown-testing” data set were correctly rejected. When applying the same threshold boundaries of 9.48 to 10.63 to Variant I and II, utilizing skeleton stacking and the Hausdorff method, both reported between four and six false positives and no false negatives.

Variant II, which utilizes the CNN besides the shape comparison algorithms, increased the accuracy in any case, except for the Jaccard-distance shape comparison algorithm without the application of skeleton stacking. A single false positive made the difference between Variant II and Variant III, when the Hausdorff distance was applied without the stacking of skeletons.

As a consequence, the combination of CNNs with the traditional shape comparison algorithms is beneficial. They work in general best, if the shape comparison algorithm first performs a rejection (Variant II), but can exceed their performance, if this process is switched (Variant III). In any case, shape comparison algorithms are useful to tweak the performance of deep learning methods. CNNs also states a certainty value besides its predictions, which can be utilized as a rejection criterion similarly to the shape comparisons. From our data, we saw, that for a learned skeleton as an input, the certainty very often was above 90 % and many times close to 100 %. In contrast, unknown data, often ranged in a certainty interval of 40 % to 50 %. The feasibility of certainty values needs to be investigated in future research.