Demographic effects on estimates of automatic face recognition performance

Abstract

The intended applications of automatic face recognition systems include venues that vary widely in demographic diversity. Formal evaluations of algorithms do not commonly consider the effects of population diversity on performance. We document the effects of racial and gender demographics on estimates of the accuracy of algorithms that match identity in pairs of face images. In particular, we focus on the effects of the “background” population distribution of non-matched identities against which identity matches are compared. The algorithm we tested was created by fusing three of the top performers from a recent US Government competition. First, we demonstrate the variability of algorithm performance estimates when the population of non-matched identities was demographically “yoked" by race and/or gender (i.e., “yoking” constrains non-matched pairs to be of the same race or gender). We also report differences in the match threshold required to obtain a false alarm rate of .001 when demographic controls on the non-matched identity pairs varied. In a second experiment, we explored the effect on algorithm performance of progressively increasing population diversity. We found systematic, but non-general, effects when the balance between majority and minority populations of non-matched identities shifted. Third, we show that identity match accuracy differs substantially when the non-match identity population varied by race. Finally, we demonstrate the impact on performance when the non-match distribution consists of faces chosen to resemble a target face. The results from all experiments indicate the importance of the demographic composition and modeling of the background population in predicting the accuracy of face recognition algorithms.

Introduction

The appearance of a face is determined by its gender, race/ethnicity, age, and identity. Any given image of a face depends also on the viewing angle, illumination, and resolution of the sensor. The goal of most face recognition algorithms is to identify someone as a unique individual. Often this test requires the algorithm to match the identity of faces between images that may vary in the quality or nature of the viewing conditions. The diversity of faces in the real world means that face recognition algorithms must operate across a backdrop of appearance variability that is not related to an individual's unique identity. Thus, face recognition algorithms intended for real-world applications must perform predictably over changes in the demographic composition of the intended application populations. The most likely application sites for algorithms include airports, border crossings, and crowded city sites such as train and metro stations. These locations are characterized by ethnically diverse populations that may vary by the time of year (e.g., tourist season) or even by the time of day (e.g., flights from the Far East arrive in the morning and European flights in the afternoon).

The performance of state-of-the-art automatic face recognition algorithms has been tested extensively over the last two decades in a series of U.S. Government-sponsored tests (e.g., [1]). Measures of algorithm performance in these tests provide the best publicly available information for making decisions about the suitability and readiness of algorithms for security and surveillance applications of importance. Traditionally, these tests have emphasized measuring performance over photometric conditions such as illumination and image resolution [2], [3], [4], [5]. They have also concentrated primarily on the quality of the “match” (i.e., similarity) between images of the same individuals, considering this to be the critical factor determining algorithm performance. When the degree of similarity between matched identities is high, as is the case when photometric factors are controlled, the algorithm is expected to perform well.

Much less consideration has been given to the distribution against which matched identities are compared. To digress briefly, all measures of the performance of face recognition algorithms rely both on the distribution of data for the population of identity matches (i.e., pairs of images of the same person) and on the distribution of data for non-matched identities (i.e., pairs of images of different people). By definition, the identity match population contains image pairs of the same race and gender. The non-matched identity population, however, may be structured in several ways. To date, in most evaluations, this distribution consists of pairs of faces that have different identities and which may, or may not, be of different races or genders.

More formally, identity match decisions (same person or different people?) are generally based on a computed similarity score between two face images. If this score exceeds a threshold similarity score, the faces are judged as an “identity match" (sometimes referred to as an “identity verification”). Otherwise, the images are judged as non-matched identities. False alarm errors occur when the computed similarity score between a pair of face images of different individuals exceeds the match threshold. In most applications, the similarity cutoff threshold is set to achieve a low false alarm rate (commonly on the order of 0.001).

As noted, in many formal evaluations of algorithms, the distribution of similarity scores for the non-matched face images includes face pairs that may differ in gender, race/ethnicity, and age [1]. The inclusion of these categorically mismatched image pairs may lead to an over-estimation of an algorithm's ability to discriminate the identity of face pairs. In other words, when non-match face pairs differ on categorical variables such as race and gender, some part of the performance of the algorithms may be due to the easier problem of discriminating faces based on race or gender, rather than to the more challenging problem of recognizing individual face identities. From a theoretical point of view, any demographic factor that decreases the average similarity of the non-matched face pairs will increase the estimated performance of an algorithm.

In this study, we focus on the problem of how the demographic composition of non-match populations affects estimates of algorithm accuracy. Although demographic variables have been shown to affect both human [6], [7], [8], [9] and machine [10], [11] accuracy at recognizing faces, the effects of these variables on the non-matched identity distributions have not been studied previously. The first goal of this study was to document the effects of yoking non-match pairs according to the categorical variables of race and gender, individually, and together. By “yoking” we mean controlling the demographic variables within a non-match pair, such that both faces in the pair are of the same gender, same race, or same gender and race. We also examine the implications of demographic control of the non-match pairs for the choice of a threshold for match/non-match decisions. The second goal was to examine algorithm accuracy with systematic variations in the proportion of a “second” ethnic group in the non-match distribution. Third, we measured algorithm accuracy when the identity matches were of a particular race and identity mismatches were from another race. This was compared to the case when the match and non-match distributions were of the same race. Finally, we look at the challenging, but plausible security application scenario, that occurs when a deliberate attempt is made to impersonate another person. Specifically, we evaluate algorithm performance estimates when the non-match distribution consists of selected imposters, chosen to be similar to a target.