A-Stacking and A-Bagging: Adaptive versions of ensemble learning algorithms for spoof fingerprint detection

doi:10.1016/j.eswa.2019.113160

Expert Systems with Applications

Volume 146, 15 May 2020, 113160

https://doi.org/10.1016/j.eswa.2019.113160 Get rights and content

Highlights

•
The behavior of stacking and bagging on spoof fingerprint detection is explored.
•
Adaptive versions of stacking and bagging are proposed.
•
Diversity is achieved by generating an ensemble of disjoint base classifiers.
•
Empirical results are provided on class balanced and imbalanced datasets.

Abstract

Stacking and bagging are widely used ensemble learning approaches that make use of multiple classifier systems. Stacking focuses on building an ensemble of heterogeneous classifiers while bagging constructs an ensemble of homogenous classifiers. There exist some applications where it is essential for learning algorithms to be adaptive towards the training data. We propose A-Stacking and A-Bagging; adaptive versions of stacking and bagging respectively that take into consideration the similarity inherently present in the dataset. One of the main motives of ensemble learning is to generate an ensemble of multiple “experts” that are weakly correlated. We achieve this by producing a set of disjoint experts where each expert is trained on a different subset of the dataset. We show the working mechanism of the proposed algorithms on spoof fingerprint detection. The proposed versions of these algorithms are adaptive as they conform to the features extracted from the live and spoof fingerprint images. From our experimental results, we establish that A-Stacking and A-Bagging give competitive results on both balanced and imbalanced datasets.

Introduction

Ensemble learning is useful in overcoming the problems of single classifier systems, i.e. computational problems: when the learning process of a weak classifier is imperfect, statistical problems: when learning data is too small to capture the entire hypotheses space and representational problems: when the true target function cannot be found by any of the hypothesis from the hypotheses space (Dietterich, 1997). One of the active areas of research in supervised learning has been to study methods for constructing good ensembles of classifiers (Dietterich, 2000a).

It has been observed that the performance of ensemble learning depends heavily on the diversity among the individual classifiers of an ensemble. Polikar (2006) defines four ways to increase the diversity among the base classifiers: 1) by using different training data to train the base classifiers, 2) by using diverse training parameters, 3) by using different features for training the base classifiers, and 4) by combining different types of classifiers.

Multiple classifier systems (MCS) sometimes referred to as a committee of classifiers or a mixture of experts have been exploited by various algorithms (Polikar, 2006). Bagging, boosting, stacking and random forest are the popular methods based on MCS paradigm. Multiple variants of these ensemble methods have been proposed and used in the past, such as Ubagging (Liang & Cohn, 2013), AdaBoost (Freund, Schapire, 1997, Sun, yue Jia, Li, 2011), AveBoost (Oza, 2003), conservative boosting (Kuncheva & Whitaker, 2002), GA-stacking (Ledezma, Aler, Sanchis, & Borrajo, 2010), cooperative ensemble learning system (CELS) (Yong Liu & Xin Yao, 1998), etc.

Stacking (Wolpert, 1992) and bagging (Breiman, 1996) are two popular ensemble learning approaches applied in various real-world scenarios such as intrusion detection, spam classification, credit scoring etc. (du Jardin, 2018, Papouskova, Hajek, 2019, Porwik, Doroz, Wrobel, 2019, Ruano-Ords, Yevseyeva, Fernandes, Mndez, Emmerich, 2019, Syarif, Zaluska, Prugel-Bennett, Wills, 2012, Zhang, Mahadevan, 2019).

Stacking uses a meta-classifier to fuse the ensemble outputs, whereas voting, weighted majority voting etc. are the common ways to combine ensemble outputs in bagging. Also, the diversity in stacking is achieved by using heterogeneous classifiers on the same training set, whereas in bagging we try to gain diversity by using the same base classifier on different training sets (Bian & Wang, 2007). However, as these different training sets are bootstrapped from a single dataset, they are not entirely disjoint with each other, which results in low diversity (Banfield, Hall, Bowyer, & Kegelmeyer, 2005).

Several modified versions of popular ensemble learning approaches have been proposed in the past (Cheplygina, Tax, Loog, 2016, Ditzler, LaBarck, Ritchie, Rosen, Polikar, 2018, Ting, Witten, 1997), but to the best of our knowledge the adaptiveness of the algorithm towards the dataset has not been explored yet.

Ensemble learning-based approaches have been used in the past for spoof fingerprint detection where the decisions of multiple base classifiers are integrated to classify an image as “live” or “spoof” (Ding, Ross, 2016, Kho, Lee, Choi, Kim, 2019). Although ensemble learning is well-known for this particular application, to the best of our knowledge, stacking has not been used for spoof fingerprint detection. We claim that for such applications, instead of straightforward usage of base classifiers, it is crucial to adapt to the features of the dataset and to adjust the learning model accordingly.

Merz (1999) argues that having a disjoint set of classifiers is advantageous in the ensemble learning as it yields weakly correlated predictions. This motivated us to maintain the diversity of the ensemble by dividing the original training set into multiple subsets using clustering. In that way, we are able to generate a diverse set of classifiers by considering the features extracted from live and spoof fingerprint images of the dataset.

The models for fingerprint recognition are vulnerable to attacks by spoof fingerprints made of different moulds of substances like silicon, wood glue, latex, gelatin, etc. Therefore, it is required to perform liveness detection before fingerprint recognition to ensure that fabricated moulds are not used for authentication. Examples of spoof fingerprints generated using these substances are shown in Fig. 1.

Local Binary Patterns (LBP) is an efficient way to determine the texture of an image by labelling each pixel with a binary value based on the thresholds on the neighbouring pixels (Jia, Yang, Cao, Zang, Zhang, Dai, Zhu, Tian, 2014, Nanni, Lumini, 2008). LBP considers the central pixel as the threshold and based on that it assigns the binary values to the neighbouring pixels. LBP value of the pixel is calculated by summing up the element-wise product of the binary values with their weights. LBP histograms are robust in terms of grayscale variations, making them suitable for spoof fingerprint detection, as they can easily incorporate fingerprints with skin distortions, different skin qualities, dry, moist or dirty skin.

•
We explore the behaviours of stacking and bagging with various base classifiers on spoof fingerprint detection problem.
•
We emphasize that the learning algorithms must be adaptive towards the properties inherent in the dataset.
•
We establish that the diversity among the ensemble of classifiers can be achieved by performing clustering on the original training set and forming subsets of it.
•
We propose adaptive models of stacking and bagging for spoof fingerprint detection and show their competitiveness on class balanced and imbalanced datasets.

Section snippets

Stacking

Stacking (Wolpert, 1992) is a learning approach based on ensemble learning which combines the predictions made by multiple base classifiers generated by using different learning algorithms $L_{1}, L_{2}, \dots L_{n}$ . These classifiers are trained on the same training data D_Train containing examples in the form $s_{i} = < x_{i}, y_{i} >,$ where x_i is the input vector, and y_i is the class label associated with it.

In the first phase, base classifiers $l_{1}, l_{2}, \dots l_{n}$ make predictions for the query instance x_q. In the second phase, the

Bagging

Bagging (Breiman, 1996) is a method of generating multiple versions of a base classifier by making bootstrapped replicates of training data and using them to get an aggregated predictor. The performance of Bagging improves if used with an unstable learner, i.e. if the learner causes significant changes by perturbing the training set.

Let the size of the original training set D_Train is N. Our task is to generate n bags of size N each by sampling D_Train with replacement. These n bags of instances

Spoof fingerprint detection

The application we consider in this paper is spoof fingerprint detection which has its importance in forensics and information security (Ding, Ross, 2016, Kho, Lee, Choi, Kim, 2019, Nogueira, de Alencar Lotufo, Campos Machado, 2016, Rattani, Scheirer, Ross, 2015). The machine learning methods for spoof/liveness detection are usually grouped into two categories: dynamic features based methods and static features based methods (Marasco & Ross, 2014). Dynamic features are identified as skin

Experimental setup

We use python Weka wrapper to use clustering and classification functionalities of Weka (Hall et al., 2009). All the original datasets have been randomized and divided into 80:20 ratio for training D_Train and validation D_Valid, so that the validation set remains disjoint from the training set. We use Simple-kMeans (Arthur & Vassilvitskii, 2007) as our clustering algorithm which performs reasonably well on the chosen datasets with k = 3. We encourage the readers to experiment with various values

Conclusions

In this study, we explore the behaviour of various ensemble learning approaches to spoof fingerprint detection. We propose A-Stacking and A-Bagging: the adaptive versions of ensemble learning approaches Stacking and Bagging, respectively. We hypothesize that the learning algorithms must take into consideration the similarity inherently present in the data. By doing so, the experts can be made adaptive towards the task associated with the dataset.

To maintain diversity among the ensemble, we

Author contribution statements

Ravindranath Chowdary C defined the problem statement and Shivang Agarwal worked on the problem under the supervision of Ravindranath Chowdary C. This work was done as part of the PhD programme of Shivang Agarwal which started in 2017. This work is our original work and is currently not submitted anywhere else.

Declaration of Competing Interest

Tha authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (56)

R.E. Banfield et al.
Ensemble diversity measures and their application to thinning
Information Fusion
(2005)
F. Barboza et al.
Machine learning models and bankruptcy prediction
Expert Systems with Applications
(2017)
I. Brown et al.
An experimental comparison of classification algorithms for imbalanced credit scoring data sets
Expert Systems with Applications
(2012)
J.-S. Chou et al.
Enhanced artificial intelligence for ensemble approach to predicting high performance concrete compressive strength
Construction and Building Materials
(2013)
Y. Freund et al.
A decision-theoretic generalization of on-line learning and an application to boosting
Journal of Computer and System Sciences
(1997)
D. Gragnaniello et al.
Local contrast phase descriptor for fingerprint liveness detection
Pattern Recognition
(2015)
P. du Jardin
Failure pattern-based ensembles applied to bankruptcy forecasting
Decision Support Systems
(2018)
X. Jia et al.
Multi-scale local binary pattern with filters for spoof fingerprint detection
Information Sciences
(2014)
J.B. Kho et al.
An incremental learning method for spoof fingerprint detection
Expert Systems with Applications
(2019)
H. Lee et al.
Neural network ensemble with probabilistic fusion and its application to gait recognition
Neurocomputing
(2009)

L. Nanni et al.

Local binary patterns for a hybrid fingerprint matcher

Pattern Recognition

(2008)

D.R. Nayak et al.

Brain mr image classification using two-dimensional discrete wavelet transform and adaboost with random forests

Neurocomputing

(2016)

M. Papouskova et al.

Two-stage consumer credit risk modelling using heterogeneous ensemble learning

Decision Support Systems

(2019)

P. Porwik et al.

An ensemble learning approach to lip-based biometric verification, with a dynamic selection of classifiers

Expert Systems with Applications

(2019)

A. Sinha et al.

Estimation and decision fusion: A survey

Neurocomputing

(2008)

I. Syarif et al.

Application of bagging, boosting and stacking to intrusion detection

G. Wang et al.

A comparative assessment of ensemble learning for credit scoring

Expert Systems with Applications

(2011)

D.H. Wolpert

Stacked generalization

Neural Networks

(1992)

Y. Xia et al.

A novel heterogeneous ensemble credit scoring model based on bstacking approach

Expert Systems with Applications

(2018)

D. Yambay et al.

Livdet 2011 - fingerprint liveness detection competition 2011

D. Arthur et al.

k-means++: the advantages of careful seeding

S. Bian et al.

On diversity and accuracy of homogeneous and heterogeneous ensembles

International Journal of Hybrid Intelligence Systems

(2007)

L. Breiman

Bagging predictors

Machine Learning

(1996)

L. Breiman

Random forests

Machine Learning

(2001)

S. le Cessie et al.

Ridge estimators in logistic regression

Applied Statistics

(1992)

V. Cheplygina et al.

Dissimilarity-based ensembles for multiple instance learning

IEEE Transactions on Neural Networks and Learning Systems

(2016)

T. Chugh et al.

IEEE Transactions on Information Forensics and Security

Fingerprint Spoof Buster: Use of Minutiae-Centered Patches

(2018)

T.G. Dietterich

Machine-learning research–four current directions

AI Magazine

(1997)

Cited by (75)

Pattern recognition system for rapid detection of gases using microfluidic olfaction detector: A case study using methane and ethane
2024, Sensors and Actuators B: Chemical
Two significant shortcomings of metal oxide semiconductor (MOS) sensors are their low selectivity and prolonged recovery time after exposure to target gases. Microfluidic olfaction detectors (MFOD), having an embedded MOS at the distal end of a microchannel, improve MOS selectivity but further prolong the recovery time due to the delay in the gas movement as the result of diffusion through the microchannel. This paper aims to reduce the sampling time of a single MFOD. Such reduction is achieved here by developing a pattern recognition system involving: (i) an ensemble of four heterogeneous base machine learning models, (ii) a Long-Short Term Memory Neural Network (LSTM) to construct a portion of the transient response, (iii) features from an additional fast response sensor, and (iv) detachment of MOS from the microchannel for accelerated recovery. Using data collected from methane and ethane as a case study, the results of classification and regression demonstrate the need for only 17.5% of the previously used response time without any loss in accuracy. This reduces the total testing cycle of an MFOD by 60%, further enhancing the ability of selective MFODs by enabling faster detection of gases.
Ensemble learning based software defect prediction
2023, Journal of Engineering Research (Kuwait)
Currently, the cost to detect and solve software defects is a heavy burden on software projects. So, it is significant to predict software defects at the earlier stages of the software development lifecycle. In this study, seven commonly-used machine learning and deep learning algorithms were studied and the performance of defect classification on 4 representative public datasets from NASA and the PROMISE repository was demonstrated. Furthermore, three classical ensemble learning methods (Bagging, Boosting, and Stacking) were used to improve the prediction performance. Six metrics, including accuracy, precision, f1-score, recall, the area under the receiver operating characteristic curve (AUC), and G-Mean were utilized to evaluate the performance. It was noted that ensemble learning exceeded all the other seven algorithms. Ensemble learning achieved the highest AUC of 0.99, the highest G-Mean of 0.96, and an average F1-score of 0.97. Under a time-sensitive scenario, the Boosting method was a good choice as it spent less runtime and had a similar performance to the other two ensemble learning methods in most cases.
Accelerating automatic hate speech detection using parallelized ensemble learning models
2023, Expert Systems with Applications
With increasing number of social media users and online engagement, it is essential to study hate speech propagation on social media platforms (SMPs). Automatic hate speech detection on social media is of utmost importance as hate speech can create discomfort among users and potentially generate a strong reaction in society. Ensemble learning algorithms are helpful in addressing sentiment-based classification due to their fault tolerance and efficiency. However, a simple, scalable, and robust framework is required to deal with large-scale data efficiently and accurately. Therefore, we propose parallelization to the standard ensemble learning algorithms to speed up the automatic hate speech detection on SMPs. In this study, we parallelize bagging, A-stacking, and random sub-space algorithms and test their serial and parallel versions on the standard high-dimensional datasets for hate speech detection. The experiments are performed using six datasets that address hate speech propagation during events like the COVID-19 pandemic, the US presidential election (2020), and the farmers’ protest in India (2021). Our parallel models observe a significant speedup with high efficiency, claiming that the proposed models are suitable for the considered application. Also, one of the main motivations of this study is to highlight the importance of generalization by testing the models under the cross-dataset environment. We observed that the accuracy is not affected while parallelizing the algorithms compared with serial algorithms executing on a single machine.
Examination of the performance of ASEL and MPViT algorithms for classifying mangrove species of multiple natural reserves of Beibu Gulf, south China
2023, Ecological Indicators
Mangrove is the highest productive ecosystems in the global coastal zone, which has high blue carbon sink function and carbon neutrality potential. Fine species classification is essential for mangrove conservation and sustainable development, and has attracted much attention in recent years using ensemble learning and multi-dimensional data. However, the current mangrove species classification based on traditional stacking ensemble learning still faces challenges due to the correlation between base classifiers, differences in meta-classifier capabilities, the subjectivity of parameter tuning, and data redundancy. To address these issues, this paper utilized unmanned aerial vehicle (UAV) multispectral images of three mangrove nature reserves in Beibu Gulf, south China, to examine the classification and generalization ability of our proposed Adaptive Stacking Ensemble Learning (ASEL) algorithm for different mangrove species. We also aim to verify the feasibility of the Multi-Path Vision Transformer for Dense Prediction (MPViT) algorithm for mangrove species mapping, and compare its performance with the ASEL algorithm for mangrove species classification. Finally, we used the SHapley Additive Explanations (SHAP) method to measure the contribution of feature variables to the model, exploring the sensitivity of different image features to mangrove species mapping. This study highlights that: (1) The two ASEL algorithms achieved high accuracy classification of mangrove species with the overall classification accuracy ranging from 79.8% to 96.2%. The ACE-Stacking and AOM-Stacking algorithms performed better classification ability than the traditional stacking algorithm, with the mean overall classification accuracy increasing from 0.9% to 3.3%. The McNemar test further indicated that the differences in classification results derived from three algorithms were significant at the 95% confidence interval, demonstrating the better classification and generalization ability of the ASEL algorithm. (2) The MPViT algorithm achieved better classification accuracy in the three reserves, with an overall accuracy of 95.5%-97.3%, which was 0.6%-5.4% higher than the two ASEL algorithms. The average accuracy of identifying mangrove species was over 95.3% in all reserves, demonstrating the desirable mangrove classification performance of MPViT algorithm. (3) The identification accuracies (F1 scores) of mangrove species in different reserves were ranged from 0.848 to 0.984. Cyperus malaccensi had the highest identification accuracy. (4) The SHAP method interpreted the great contribution of digital surface model and variable atmospherically resistant index features on mangrove species classification. The red band and ratio vegetation index was sensitive to Aegiceras corniculatum and to Cyperus malaccensi.
Uncertainty management in electricity demand forecasting with machine learning and ensemble learning: Case studies of COVID-19 in the US metropolitans
2023, Engineering Applications of Artificial Intelligence
Improving load forecasting is becoming increasingly crucial for power system management and operational research. Disruptive influences can seriously impact both the supply and demand sides of power. This work examines the impact of the coronavirus on power usage in two US states from January 2020 to December 2020. A wide range of machine learning (ML) algorithms and ensemble learning are employed to conduct the analysis. The findings showed a surprising increase in monthly power use changes in Florida and Texas during the COVID-19 pandemic, in contrast to New York, where usage decreased over the same period. In Texas, the quantity of power usage rises from 2% to 6% practically every month, except for September, when it decreased by around 1%. For Florida, except for May, which showed a fall of roughly 2.5%, the growth varied from 2.5% to 7.5%. This indicates the need for more extensive research into such systems and the applicability of adopting groups of algorithms in learning the trends of electric power demand during uncertain events. Such learning will be helpful in forecasting future power demand changes due to especially public health-related scenarios.
SFincBuster: Spoofed fingerprint buster via incremental learning using leverage bagging classifier
2023, Image and Vision Computing
The fingerprint presentation attack detection (PAD) studies are extensively explored by investigators to augment the security aspects of human authentication in biometric systems. Although existing approaches yield promising results while evaluated on data having same distribution as the training data, but their performance is declined in scenarios when data is gathered from unknown environment. In these situations model is learned with the new type of fake samples by re-training the entire model. Nevertheless, it has been a non-trivial concern to re-train the entire system to tackle the newly created fake samples as unseen artifacts are generated continuously in real-time scenarios. In this work, we expound a novel incremental learning-based approach namely; SFincBuster that can work effectively in the challenging real-time scenarios and can handle both new data as well as new observations from old classes. We train a leveraging bagging ensemble (LBE) in incremental fashion regardless of the extracted deep-level features (using pre-trained VGG19 network) being too large to accommodate into system memory, it is still possible to train our model effectively. Furthermore, the LBE integrates the simplicity of classical bagging with augmented randomization to the input and outcome of the base classifiers. To tackle the issue of change in distribution that arises with gradual changes from learned fakes to entirely new fingerprint artifacts, the SFincBuster employs LBE with ADWIN (Adaptive WINDowing) technique that continuously evaluate the performance of underlying base model and whenever a change is detected the weakest classifier is substituted with a new one. Our approach achieves high classification accuracy, even though it is not prerequisite to access all features at once. The SFincBuster is trained and evaluated on LivDet 2009, LivDet 2011, LivDet 2013, LivDet 2015 and LivDet 2021 benchmark datasets and yields maximum average classification accuracy (ACA) of 98.65% on LivDet 2013 and LivDet 2015 datasets. The model exhibits stupendous generalization capabilities with an average classification error rate (ACER) of 1.39% for known fakes (KF) and 2.84% for unknown fakes (UF). Finally, the comparable investigation perceives that the SFincBuster model demonstrates a noteworthy performance gain over the similar state-of-the-art (SOTA) approaches and achieve an improved benchmark for real-time cross-sensor, cross-material and cross-dataset scenarios.

View all citing articles on Scopus

View full text

A-Stacking and A-Bagging: Adaptive versions of ensemble learning algorithms for spoof fingerprint detection

Highlights

Abstract

Introduction

Section snippets

Stacking

Bagging

Spoof fingerprint detection

Experimental setup

Conclusions

Author contribution statements

Declaration of Competing Interest

Information Fusion

Expert Systems with Applications

Expert Systems with Applications

Construction and Building Materials

Journal of Computer and System Sciences

Pattern Recognition

Decision Support Systems

Information Sciences

Expert Systems with Applications

Neurocomputing

Pattern Recognition

Neurocomputing

Decision Support Systems

Expert Systems with Applications

Neurocomputing

Expert Systems with Applications

Neural Networks

Expert Systems with Applications

k-means++: the advantages of careful seeding

On diversity and accuracy of homogeneous and heterogeneous ensembles

International Journal of Hybrid Intelligence Systems

Bagging predictors

Machine Learning

Random forests

Machine Learning

Ridge estimators in logistic regression

Applied Statistics

Dissimilarity-based ensembles for multiple instance learning

IEEE Transactions on Neural Networks and Learning Systems

IEEE Transactions on Information Forensics and Security

Fingerprint Spoof Buster: Use of Minutiae-Centered Patches

Machine-learning research–four current directions

AI Magazine