Sensitivity analysis for biometric systems: A methodology based on orthogonal experiment designs☆
Highlights
► Introduce an effective/structured sensitivity analysis methodology for biometric systems. ► Apply an orthogonal experiment design and statistical data analysis (graphical procedures). ► Provide insight and understanding of the key factors that affect the system performance. ► Illustrate the methodology by application to VASIR (NIST iris-based biometric system). ► Determine important factors, important interactions, and local best settings.
Introduction
Biometrics is the automated recognition of individuals based on their biological and behavioral characteristics [1]. The characteristics can include fingerprints, face, iris, ocular area, retina, ear, voice, DNA, signature, gait, and hand geometry among others. The use of biometrics has many advantages—especially as an alternative to keys, passwords, smartcards, and other artifacts for physical entry. In this regard, biometric-based technologies are increasingly being incorporated into specific security fields and applications, such as industrial access control, law-enforcement, military, border control, and forensics [2].
A significant problem in biometric studies is that researchers/developers often present results that lack an assessment of intrinsic system uncertainty. A high degree of input and output numerical precision often gives the impression of great accuracy, but neglects to give attention to the critical questions of the sensitivity of the final results to different algorithms, environments, subject characteristics, and biometric sample conditions [3]. Hornberger and Spear [4] made the following paraphrased statement about simulation models: Most such models are complex, with many parameters, state-variables and underlying non-linear relations; under optimal circumstances, such systems have many degrees of freedom and—with judicious adjustments—are susceptible to over-fitting with both plausible structure and “reasonable” parameter values. We believe that the above statement applies equally well for biometric systems, especially for iris recognition system.
Sensitivity analysis has been successfully conducted in areas such as computer vision and computer network [5], [6], [7]. Sensitivity analysis is the study of how the output of a system is affected by different inputs to the system [8]. In essence, a biometric system is a data monitoring and decision-making “machine.” A good biometric system has a high proportion of correct decisions. All biometric systems are susceptible to incorrect decisions—especially in the presence of less-than-optimal conditions.
In practice, the performance of many biometric systems is frequently examined and optimized via a series of one-factor-at-a-time experiment designs in which most factors in the system are held constant while one factor is focused on and varied to examine its effect. This design, though popular [9], [10] has the disadvantage that it can yield grossly biased (incorrect) estimates of factor effects. Further, this design has no capacity to estimate factor interactions—which are intrinsic to many biometric systems.
The motivation for this paper is thus to introduce to the biometrics community an alternative method for conducting a sensitivity analysis with the advantage that:
- (1)
The system will be better understood.
- (2)
The system will be better characterized.
- (3)
The system will be better optimized—with the net effect that system performance is significantly improved in a computationally efficient fashion.
Thus, in short, the objective of this paper is to introduce and apply a structured “Sensitivity Analysis” approach for gaining insight and understanding about the system’s key components—those which most affect the quality and performance of a biometric system—and to optimize the settings of these key components.
Sensitivity analysis as we describe it has two separate and distinct steps:
- (1)
Experiment Design (the structured plan for collecting the data), and
- (2)
Statistical Analysis (the structured methodology for analyzing the data).
Both parts are critical, and when optimally used in concert yield enhanced insight into the relative importance and effect of the various computational components (and interactions) affecting biometric system performance. The experiment design and data analysis are demonstrated by a particular iris-based biometric system, VASIR (Video-based Automated System for Iris Recognition), which has verification capability for both traditional still iris images and video sequences captured at a distance while a person walks through a portal [11].
The general structure of this paper is threefold:
- (1)
Orthogonal fractional factorial design: Introduce to the biometrics community a structured orthogonal fractional factorial experiment design methodology to efficiently gain insight and understanding (“sensitivity analysis”) of critical system parameters, interactions, and their optimal settings—this introduces and applies an established method within the statistical community [12], [13].
- (2)
Statistical analysis: Present effective and insightful statistical analysis methodologies for carrying out sensitivity studies.
- (3)
Demonstration with VASIR: Demonstrate our experiment design and analysis methodologies for VASIR, with potential application to the broader biometrics field.
This sensitivity analysis approach provides a tool for understanding the computational components affecting the overall performance of a biometric system. Based on such understanding, the logical follow-up is to carry out an optimization analysis (identifying the optimal global settings of the components), and a robustness analysis (assessing the range of validity for our sensitivity and optimization conclusions). Our current paper focuses on the details of the sensitivity analysis only. To demonstrate the elements of the sensitivity analysis approach, we restrict our focus to a fixed setting for two robustness factors: (1) eye position (left eye only), and (2) image type (video matching: non-ideal to non-ideal image only).
Section snippets
Sensitivity analysis methodology
Sensitivity analysis is the experimental process by which we determine the relative importance of the various factors of a system. Suppose a system has k factors (input parameters) which potentially affect its performance. The minimal deliverable of a sensitivity analysis is to produce a ranked list of those k factors—ordered most to least important. For complicated systems (e.g., biometrics), a more desirable deliverable is to produce a ranked list which contains not only the k main factors,
VASIR (Video-based Automatic System for Iris Recognition)
Iris recognition is a popular biometric system approach whose effectiveness is due to the highly distinctive features of the human iris. Most commercial systems for iris recognition are relatively expensive and are computational black boxes that run proprietary algorithms. In this light, to advance iris-based biometrics technology—IrisBEE (Iris Biometric Evaluation Environment) [19] algorithm—was implemented in the C programming language from Masek’s Matlab code [9]. IrisBEE was developed as a
Experiment design: VASIR case
The purpose of a Sensitivity Analysis is first and foremost to gain insight into the important factors (and interactions) which drive the biometrics system. In this regard, the primary Sensitivity Analysis output is a ranked list of factors and interactions along with estimates of the magnitude of their effects. To achieve that, the biometrics researcher needs to provide information about the following:
- (1)
model;
- (2)
factors;
- (3)
responses;
- (4)
max affordable number of runs; and
- (5)
choice of design.
Dataset
For the purpose of this sensitivity analysis study, we evaluated the VASIR system performance using datasets collected by MBGC (Multiple Biometric Grand Challenge) [39]. These MBGC datasets include iris images of varying illumination conditions, low quality, and off-angle or occluded images in both still and video imagery. One of the challenges for the MBGC dataset is to recognize a person using an iris from the NIR and high definition video as the person walks through a portal. In this
Data analysis
This section describes the details of the sensitivity analysis carried out on the k = 8 algorithmic factors for the VASIR system. This analysis is carried out and presented for the fixed settings of the remaining algorithmic factors (see Appendix A) and robustness factors (see Table 1). In particular, the results presented in the remainder of this section are for the VV matching scenario and the left eye position.
The specific deliverables from the sensitivity analysis of the VASIR system are as
Results
This section is a summary of the results for the factor effects, most important to least important factors, optimal settings, and performance improvement.
For this VV Left case, the ranked list of factors is (X2, X7, X3, X1, X8, X4, X6, X5) of which six factors: X2 through X4 are statistically significant; factors X6 and X5 are not statistically significant. Factor X2 (IQMetr) is image quality metrics and factor X7 (SMAlg) is similarity metrics—these are the two most important algorithmic factors
Conclusions
We introduced to the biometrics community a structured methodology for sensitivity analysis to foster an understanding of the key factors (parameters) in biometric systems.
This Sensitivity Analysis methodology consists of two components:
- (1)
Experiment design in which we utilize efficient orthogonal fractional factorial designs to estimate not only the k main effects but also the two-term interactions of a biometric system.
- (2)
Graphical data analysis in which we utilize three procedures: Main Effects
Disclaimer
The identification of any commercial product or trade name does not imply endorsement or recommendation by the National Institute of Standards and Technology (NIST).
Yooyoung Lee is a guest researcher at NIST and primary focuses on video-based biometrics and evaluations. She received her Ph.D. in Computer Engineering from Chung-Ang University, Seoul, Korea. She received the Associate of the Year (2012) Award from the Information Technology Laboratory (ITL) at NIST for analyzing and evaluating the performance of biometric algorithms. She developed Video-based Automatic System for Iris Recognition (VASIR) as a benchmark baseline. Her research interests are
References (42)
- et al.
A sensitivity analysis method and its application in physics-based nonrigid motion modeling
Image Vision Comput.
(2007) - et al.
On the sensitivity analysis of camera calibration from images of spheres
Comput. Vision Image Understand.
(2010) - et al.
Fusing color and shape descriptors in the recognition of degraded iris images acquired at visible wavelengths
Comput. Vision Image Understand.
(2012) - ISO/IEC 2382-37:2012(E) Information technology—Vocabulary—Part 37: Biometrics, International Organization for...
- A.K. Jain, P.J. Flynn, A.A. Ross, Introduction to biometrics, in: Handbook of Biometrics,...
- G. Stirling Andrew, On science and precaution in the management of technological risk, Eur. Sci. Technol. Observatory...
- G.M. Hornberger, S.R.C., An approach to the preliminary analysis of environmental systems, J. Environ. Manage. 12...
- K.L. Mills, J.J. Filliben, C. Dong Yeon, S. Edward, G. Daniel, Sensitivity Analysis of MesoNet, Study of Proposed...
Sensitivity analysis for importance assessment
Risk Anal.
(2002)- L. Masek, P. Kovesi, MATLAB Source Code for a Biometric Identification System based on Iris Patterns, School of...
Exploratory Data Analysis Techniques as Applied to a High-Precision Turning Machine
Design and Analysis of Monte Carlo Experiments
Handbook of Computational Statistics
Statistics for Experimenters: An Introduction to Design, and Model Building
DATAPLOT—an interactive high-level language for graphics, non-linear fitting, data analysis, and mathematics
ACM SIGGRAPH Comput. Graph.
An automated video-based system for iris recognition
Adv. Biomet.
Cited by (0)
Yooyoung Lee is a guest researcher at NIST and primary focuses on video-based biometrics and evaluations. She received her Ph.D. in Computer Engineering from Chung-Ang University, Seoul, Korea. She received the Associate of the Year (2012) Award from the Information Technology Laboratory (ITL) at NIST for analyzing and evaluating the performance of biometric algorithms. She developed Video-based Automatic System for Iris Recognition (VASIR) as a benchmark baseline. Her research interests are video-based human recognition, biometrics, performance evaluations, computer vision, pattern recognition, image processing, and sensitivity analysis.
James J. Filliben is a mathematical statistician at NIST. He received his Ph.D. in statistics from Princeton University (1969). He is the author of more than 100 papers, the developer of a widely-cited test statistic for normality, a Fellow of the American Statistical Society, and the 2003 recipient of the ASA Youden Award (Interlab Testing). He is the author of Dataplot: an extensive software system for statistical graphics and modeling. His NIST and inter-agency contributions have been recognized with Department of Commerce Gold (3), Silver (1), and Bronze (3) medals. His research interests include exploratory data analysis, statistical graphics, distributional modeling, and experiment design.
Ross J. Micheals leads an effort researching biometric client technologies with a particular focus on web services & usable next-generation interfaces for acquisition. Ross also pioneered NIST’s research into the usability of biometric systems, an initiative in which he is still involved with today. Most recently, he served as the acting deputy director for the National Strategy for Trusted Identities in Cyberspace (NSTIC) Program Office. He earned his Ph.D. in computer science from Lehigh University, and has also worked in the field of computer vision for both Texas Instruments and Carnegie Mellon’s Robotics Institute. Since 1998, he has collaborated with 30 co-authors and has been cited by nearly 600 researchers. Along with his colleagues, he was awarded a Department of Commerce Gold Medal in 2003 for his work on biometric system performance assessment.
Jonathon Phillips is a leading technologist in the fields of computer vision, biometrics, and face recognition. He is at NIST, where he runs challenge problems and evaluations to advance biometric technology. His previous efforts include the Iris Challenge Evaluations (ICE), the Face Recognition Vendor Test (FRVT) 2006 and the Face Recognition Grand Challenge and FERET. From 2000-2004, Dr. Phillips was assigned to DARPA. For his work on the FRVT 2002 he was awarded the Department of Commerce Gold Medal. His work has been reported in the New York Times and the Economist. He has appeared on NPR’s Science Friday show. He is a fellow of the IEEE and IAPR.
- ☆
This paper has been recommended for acceptance by Rudolf M. Bolle.