Sensitivity analysis for biometric systems: A methodology based on orthogonal experiment designs

https://doi.org/10.1016/j.cviu.2013.01.003Get rights and content

Abstract

The purpose of this paper is to introduce an effective and structured methodology for carrying out a biometric system sensitivity analysis. The goal of sensitivity analysis is to provide the researcher/developer with insight and understanding of the key factors—algorithmic, subject-based, procedural, image quality, environmental, among others—that affect the matching performance of the biometric system under study. This proposed methodology consists of two steps: (1) the design and execution of orthogonal fractional factorial experiment designs which allow the scientist to efficiently investigate the effect of a large number of factors—and interactions—simultaneously, and (2) the use of a select set of statistical data analysis graphical procedures which are fine-tuned to unambiguously highlight important factors, important interactions, and locally-optimal settings. We illustrate this methodology by application to a study of VASIR (Video-based Automated System for Iris Recognition)—NIST iris-based biometric system. In particular, we investigated k = 8 algorithmic factors from the VASIR system by constructing a (26−1 × 31  × 41) orthogonal fractional factorial design, generating the corresponding performance data, and applying an appropriate set of analysis graphics to determine the relative importance of the eight factors, the relative importance of the 28 two-term interactions, and the local best settings of the eight algorithms. The results showed that VASIR’s performance was primarily driven by six factors out of the eight, along with four two-term interactions. A virtue of our two-step methodology is that it is systematic and general, and hence may be applied with equal rigor and effectiveness to other biometric systems, such as fingerprints, face, voice, and DNA.

Highlights

► Introduce an effective/structured sensitivity analysis methodology for biometric systems. ► Apply an orthogonal experiment design and statistical data analysis (graphical procedures). ► Provide insight and understanding of the key factors that affect the system performance. ► Illustrate the methodology by application to VASIR (NIST iris-based biometric system). ► Determine important factors, important interactions, and local best settings.

Introduction

Biometrics is the automated recognition of individuals based on their biological and behavioral characteristics [1]. The characteristics can include fingerprints, face, iris, ocular area, retina, ear, voice, DNA, signature, gait, and hand geometry among others. The use of biometrics has many advantages—especially as an alternative to keys, passwords, smartcards, and other artifacts for physical entry. In this regard, biometric-based technologies are increasingly being incorporated into specific security fields and applications, such as industrial access control, law-enforcement, military, border control, and forensics [2].

A significant problem in biometric studies is that researchers/developers often present results that lack an assessment of intrinsic system uncertainty. A high degree of input and output numerical precision often gives the impression of great accuracy, but neglects to give attention to the critical questions of the sensitivity of the final results to different algorithms, environments, subject characteristics, and biometric sample conditions [3]. Hornberger and Spear [4] made the following paraphrased statement about simulation models: Most such models are complex, with many parameters, state-variables and underlying non-linear relations; under optimal circumstances, such systems have many degrees of freedom and—with judicious adjustments—are susceptible to over-fitting with both plausible structure and “reasonable” parameter values. We believe that the above statement applies equally well for biometric systems, especially for iris recognition system.

Sensitivity analysis has been successfully conducted in areas such as computer vision and computer network [5], [6], [7]. Sensitivity analysis is the study of how the output of a system is affected by different inputs to the system [8]. In essence, a biometric system is a data monitoring and decision-making “machine.” A good biometric system has a high proportion of correct decisions. All biometric systems are susceptible to incorrect decisions—especially in the presence of less-than-optimal conditions.

In practice, the performance of many biometric systems is frequently examined and optimized via a series of one-factor-at-a-time experiment designs in which most factors in the system are held constant while one factor is focused on and varied to examine its effect. This design, though popular [9], [10] has the disadvantage that it can yield grossly biased (incorrect) estimates of factor effects. Further, this design has no capacity to estimate factor interactions—which are intrinsic to many biometric systems.

The motivation for this paper is thus to introduce to the biometrics community an alternative method for conducting a sensitivity analysis with the advantage that:

  • (1)

    The system will be better understood.

  • (2)

    The system will be better characterized.

  • (3)

    The system will be better optimized—with the net effect that system performance is significantly improved in a computationally efficient fashion.

Thus, in short, the objective of this paper is to introduce and apply a structured “Sensitivity Analysis” approach for gaining insight and understanding about the system’s key components—those which most affect the quality and performance of a biometric system—and to optimize the settings of these key components.

Sensitivity analysis as we describe it has two separate and distinct steps:

  • (1)

    Experiment Design (the structured plan for collecting the data), and

  • (2)

    Statistical Analysis (the structured methodology for analyzing the data).

Both parts are critical, and when optimally used in concert yield enhanced insight into the relative importance and effect of the various computational components (and interactions) affecting biometric system performance. The experiment design and data analysis are demonstrated by a particular iris-based biometric system, VASIR (Video-based Automated System for Iris Recognition), which has verification capability for both traditional still iris images and video sequences captured at a distance while a person walks through a portal [11].

The general structure of this paper is threefold:

  • (1)

    Orthogonal fractional factorial design: Introduce to the biometrics community a structured orthogonal fractional factorial experiment design methodology to efficiently gain insight and understanding (“sensitivity analysis”) of critical system parameters, interactions, and their optimal settings—this introduces and applies an established method within the statistical community [12], [13].

  • (2)

    Statistical analysis: Present effective and insightful statistical analysis methodologies for carrying out sensitivity studies.

  • (3)

    Demonstration with VASIR: Demonstrate our experiment design and analysis methodologies for VASIR, with potential application to the broader biometrics field.

This sensitivity analysis approach provides a tool for understanding the computational components affecting the overall performance of a biometric system. Based on such understanding, the logical follow-up is to carry out an optimization analysis (identifying the optimal global settings of the components), and a robustness analysis (assessing the range of validity for our sensitivity and optimization conclusions). Our current paper focuses on the details of the sensitivity analysis only. To demonstrate the elements of the sensitivity analysis approach, we restrict our focus to a fixed setting for two robustness factors: (1) eye position (left eye only), and (2) image type (video matching: non-ideal to non-ideal image only).

Section snippets

Sensitivity analysis methodology

Sensitivity analysis is the experimental process by which we determine the relative importance of the various factors of a system. Suppose a system has k factors (input parameters) which potentially affect its performance. The minimal deliverable of a sensitivity analysis is to produce a ranked list of those k factors—ordered most to least important. For complicated systems (e.g., biometrics), a more desirable deliverable is to produce a ranked list which contains not only the k main factors,

VASIR (Video-based Automatic System for Iris Recognition)

Iris recognition is a popular biometric system approach whose effectiveness is due to the highly distinctive features of the human iris. Most commercial systems for iris recognition are relatively expensive and are computational black boxes that run proprietary algorithms. In this light, to advance iris-based biometrics technology—IrisBEE (Iris Biometric Evaluation Environment) [19] algorithm—was implemented in the C programming language from Masek’s Matlab code [9]. IrisBEE was developed as a

Experiment design: VASIR case

The purpose of a Sensitivity Analysis is first and foremost to gain insight into the important factors (and interactions) which drive the biometrics system. In this regard, the primary Sensitivity Analysis output is a ranked list of factors and interactions along with estimates of the magnitude of their effects. To achieve that, the biometrics researcher needs to provide information about the following:

  • (1)

    model;

  • (2)

    factors;

  • (3)

    responses;

  • (4)

    max affordable number of runs; and

  • (5)

    choice of design.

Dataset

For the purpose of this sensitivity analysis study, we evaluated the VASIR system performance using datasets collected by MBGC (Multiple Biometric Grand Challenge) [39]. These MBGC datasets include iris images of varying illumination conditions, low quality, and off-angle or occluded images in both still and video imagery. One of the challenges for the MBGC dataset is to recognize a person using an iris from the NIR and high definition video as the person walks through a portal. In this

Data analysis

This section describes the details of the sensitivity analysis carried out on the k = 8 algorithmic factors for the VASIR system. This analysis is carried out and presented for the fixed settings of the remaining algorithmic factors (see Appendix A) and robustness factors (see Table 1). In particular, the results presented in the remainder of this section are for the VV matching scenario and the left eye position.

The specific deliverables from the sensitivity analysis of the VASIR system are as

Results

This section is a summary of the results for the factor effects, most important to least important factors, optimal settings, and performance improvement.

For this VV Left case, the ranked list of factors is (X2, X7, X3, X1, X8, X4, X6, X5) of which six factors: X2 through X4 are statistically significant; factors X6 and X5 are not statistically significant. Factor X2 (IQMetr) is image quality metrics and factor X7 (SMAlg) is similarity metrics—these are the two most important algorithmic factors

Conclusions

We introduced to the biometrics community a structured methodology for sensitivity analysis to foster an understanding of the key factors (parameters) in biometric systems.

This Sensitivity Analysis methodology consists of two components:

  • (1)

    Experiment design in which we utilize efficient orthogonal fractional factorial designs to estimate not only the k main effects but also the k2 two-term interactions of a biometric system.

  • (2)

    Graphical data analysis in which we utilize three procedures: Main Effects

Disclaimer

The identification of any commercial product or trade name does not imply endorsement or recommendation by the National Institute of Standards and Technology (NIST).

Yooyoung Lee is a guest researcher at NIST and primary focuses on video-based biometrics and evaluations. She received her Ph.D. in Computer Engineering from Chung-Ang University, Seoul, Korea. She received the Associate of the Year (2012) Award from the Information Technology Laboratory (ITL) at NIST for analyzing and evaluating the performance of biometric algorithms. She developed Video-based Automatic System for Iris Recognition (VASIR) as a benchmark baseline. Her research interests are

References (42)

  • K. Messer, J. Kittler, M. Sadeghi, M. Hamouz, A. Kostyn, S. Marcel, S. Bengio, F. Cardinaux, C. Sanderson, N. Poh, et...
  • Y. Lee, VASIR: Video-based Automatic System for Iris Recognition, Dissertation, School of Computer Science and...
  • J.J. Filliben et al.

    Exploratory Data Analysis Techniques as Applied to a High-Precision Turning Machine

    (1993)
  • J.J. Filliben, Experiment Design for Scientists and Engineers, National Institute of Standards and Technology,...
  • National Institute of Standards and Technology, Engineering Statistics HandBook (5.5.9. An EDA Approach to Experimental...
  • J.P. Kleijnen

    Design and Analysis of Monte Carlo Experiments

    Handbook of Computational Statistics

    (2004)
  • G.E.P. Box et al.

    Statistics for Experimenters: An Introduction to Design, and Model Building

    (1978)
  • J.J. Filliben, A. Hecket, Dataplot Homepage, <http://www.itl.nist.gov/div898/software/dataplot/> (accessed...
  • J.J. Filliben

    DATAPLOT—an interactive high-level language for graphics, non-linear fitting, data analysis, and mathematics

    ACM SIGGRAPH Comput. Graph.

    (1981)
  • P.J. Phillips, K.W. Bowyer, P.J. Flynn, X. Liu, W.T. Scruggs, The iris challenge evaluation 2005, in: 2nd IEEE...
  • Y. Lee et al.

    An automated video-based system for iris recognition

    Adv. Biomet.

    (2009)
  • Cited by (0)

    Yooyoung Lee is a guest researcher at NIST and primary focuses on video-based biometrics and evaluations. She received her Ph.D. in Computer Engineering from Chung-Ang University, Seoul, Korea. She received the Associate of the Year (2012) Award from the Information Technology Laboratory (ITL) at NIST for analyzing and evaluating the performance of biometric algorithms. She developed Video-based Automatic System for Iris Recognition (VASIR) as a benchmark baseline. Her research interests are video-based human recognition, biometrics, performance evaluations, computer vision, pattern recognition, image processing, and sensitivity analysis.

    James J. Filliben is a mathematical statistician at NIST. He received his Ph.D. in statistics from Princeton University (1969). He is the author of more than 100 papers, the developer of a widely-cited test statistic for normality, a Fellow of the American Statistical Society, and the 2003 recipient of the ASA Youden Award (Interlab Testing). He is the author of Dataplot: an extensive software system for statistical graphics and modeling. His NIST and inter-agency contributions have been recognized with Department of Commerce Gold (3), Silver (1), and Bronze (3) medals. His research interests include exploratory data analysis, statistical graphics, distributional modeling, and experiment design.

    Ross J. Micheals leads an effort researching biometric client technologies with a particular focus on web services & usable next-generation interfaces for acquisition. Ross also pioneered NIST’s research into the usability of biometric systems, an initiative in which he is still involved with today. Most recently, he served as the acting deputy director for the National Strategy for Trusted Identities in Cyberspace (NSTIC) Program Office. He earned his Ph.D. in computer science from Lehigh University, and has also worked in the field of computer vision for both Texas Instruments and Carnegie Mellon’s Robotics Institute. Since 1998, he has collaborated with 30 co-authors and has been cited by nearly 600 researchers. Along with his colleagues, he was awarded a Department of Commerce Gold Medal in 2003 for his work on biometric system performance assessment.

    Jonathon Phillips is a leading technologist in the fields of computer vision, biometrics, and face recognition. He is at NIST, where he runs challenge problems and evaluations to advance biometric technology. His previous efforts include the Iris Challenge Evaluations (ICE), the Face Recognition Vendor Test (FRVT) 2006 and the Face Recognition Grand Challenge and FERET. From 2000-2004, Dr. Phillips was assigned to DARPA. For his work on the FRVT 2002 he was awarded the Department of Commerce Gold Medal. His work has been reported in the New York Times and the Economist. He has appeared on NPR’s Science Friday show. He is a fellow of the IEEE and IAPR.

    This paper has been recommended for acceptance by Rudolf M. Bolle.

    View full text