Abstract
The goal of information security is to prevent unauthorized access to data. There are several conventional ways to confirm user identity, such as using a password, user name, and keys. These conventional methods are rather limited; they can be stolen, lost, copied, or cracked. Because multimodal biometric identification systems are more secure and have higher recognition efficiency than unimodal biometric systems, they get attention. Single-modal biometric recognition systems perform poorly in real-world public security operations because of poor biometric data quality. Some of the drawbacks of current multimodal fusion methods include low generalization and single-level fusion. This study presents a novel multimodal biometric fusion model that significantly enhances accuracy and generalization through the power of artificial intelligence. Various fusion methods, encompassing pixel-level, feature-level, and score-level fusion, are seamlessly integrated through deep neural networks. At the pixel level, we employ spatial, channel, and intensity fusion strategies to optimize the fusion process. On the feature level, modality-specific branches and jointly optimized representation layers establish robust dependencies between modalities through backpropagation. Finally, intelligent fusion techniques, such as Rank-1 and modality evaluation, are harnessed to blend matching scores on the score level. To validate the model's effectiveness, we construct a virtual homogeneous multimodal dataset using simulated operational data. Experimental results showcase significant improvements compared to single-modal algorithms, with a remarkable 2.2 percentage point increase in accuracy achieved through multimodal feature fusion. The score fusion method surpasses single-modal algorithms by 3.5 percentage points, reaching an impressive retrieval accuracy of 99.6%.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-024-18509-0/MediaObjects/11042_2024_18509_Fig10_HTML.png)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
Data shall be made available on request.
References
Ö. Toygar, F. O. Babalola and Y. Bi̇ti̇ri̇m (2020) "FYO: A Novel Multimodal Vein Database With Palmar, Dorsal and Wrist Biometrics," in IEEE Access 8: 82461–8247. https://doi.org/10.1109/ACCESS.2020.2991475.
Kumar P, Mukherjee S, Saini R, Kaushik P, Roy PP, Dogra DP (2019) Multimodal Gait Recognition With Inertial Sensor Data and Video Using Evolutionary Algorithm. IEEE Trans Fuzzy Syst 27(5):956–965. https://doi.org/10.1109/TFUZZ.2018.2870590
Atenco JC, Moreno JC, Ramírez JM (2023) Deep Learning Convolutional Network for Bimodal Biometric Recognition with Information Fusion at Feature Level. IEEE Lat Am Trans 21(5):652–661. https://doi.org/10.1109/TLA.2023.10130837
Yuan C, Jiao S, Sun X, Wu QMJ (2022) MFFFLD: A Multimodal-Feature-Fusion-Based Fingerprint Liveness Detection. IEEE Transactions on Cognitive and Developmental Systems 14(2):648–661. https://doi.org/10.1109/TCDS.2021.3062624
Huang Y, Ma H, Wang M (2023) Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity. IEEE Access 11:17497–17509. https://doi.org/10.1109/ACCESS.2023.3242984
Hammad M, Liu Y, Wang K (2019) Multimodal Biometric Authentication Systems Using Convolution Neural Network Based on Different Level Fusion of ECG and Fingerprint. IEEE Access 7:26527–26542. https://doi.org/10.1109/ACCESS.2018.2886573
Kanhangad V, Kumar A, Zhang D (2008) "Comments on “An Adaptive Multimodal Biometric Management Algorithm,” in IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 38(6):841–843. https://doi.org/10.1109/TSMCC.2008.2001570
Poh N et al (2009) Benchmarking Quality-Dependent and Cost-Sensitive Score-Level Multimodal Biometric Fusion Algorithms. IEEE Trans Inf Forensics Secur 4(4):849–866. https://doi.org/10.1109/TIFS.2009.2034885
Snelick R, Uludag U, Mink A, Indovina M, Jain A (2005) Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems. IEEE Trans Pattern Anal Mach Intell 27(3):450–455. https://doi.org/10.1109/TPAMI.2005.57
Poh N, Kittler J, Bourlai T (2010) Quality-Based Score Normalization With Device Qualitative Information for Multimodal Biometric Fusion. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 40(3):539–554. https://doi.org/10.1109/TSMCA.2010.2041660
Shekhar S, Patel VM, Nasrabadi NM, Chellappa R (2014) Joint Sparse Representation for Robust Multimodal Biometrics Recognition. IEEE Trans Pattern Anal Mach Intell 36(1):113–126. https://doi.org/10.1109/TPAMI.2013.109
Walia GS, Jain G, Bansal N, Singh K (2020) Adaptive Weighted Graph Approach to Generate Multimodal Cancelable Biometric Templates. IEEE Trans Inf Forensics Secur 15:1945–1958. https://doi.org/10.1109/TIFS.2019.2954779
Monwar MM, Gavrilova ML (2009) “Multimodal Biometric System Using Rank-Level Fusion Approach,” in IEEE Transactions on Systems, Man, and Cybernetics. Part B (Cybernetics) 39(4):867–878. https://doi.org/10.1109/TSMCB.2008.2009071
Haghighat M, Abdel-Mottaleb M, Alhalabi W (2016) Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition. IEEE Trans Inf Forensics Secur 11(9):1984–1996. https://doi.org/10.1109/TIFS.2016.2569061
Nguyen K, Denman S, Sridharan S, Fookes C (2015) Score-Level Multibiometric Fusion Based on Dempster-Shafer Theory Incorporating Uncertainty Factors. IEEE Transactions on Human-Machine Systems 45(1):132–140. https://doi.org/10.1109/THMS.2014.2361437
Conti V, Militello C, Sorbello F, Vitabile S (2010) “A Frequency-based Approach for Features Fusion in Fingerprint and Iris Multimodal Biometric Identification Systems,” in IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 40(4):384–395. https://doi.org/10.1109/TSMCC.2010.2045374
Poh N, Windridge D, Mottl V, Tatarchuk A, Eliseyev A (2010) Addressing Missing Values in Kernel-Based Multimodal Biometric Fusion Using Neutral Point Substitution. IEEE Trans Inf Forensics Secur 5(3):461–469. https://doi.org/10.1109/TIFS.2010.2053535
Zhang X, Cheng D, Jia P, Dai Y, Xu X (2020) An Efficient Android-Based Multimodal Biometric Authentication System With Face and Voice. IEEE Access 8:102757–102772. https://doi.org/10.1109/ACCESS.2020.2999115
Guo BH, Nixon MS, Carter JN (2019) Soft Biometric Fusion for Subject Recognition at a Distance. IEEE Trans Biomet Behav Identity Sci 1(4):292–301. https://doi.org/10.1109/TBIOM.2019.2943934
Parashar A, Parashar A, Abate AF, Shekhawat RS, Rida I (2023) Real-time gait biometrics for surveillance applications: A review. In Image and Vision Computing 138:104784. https://doi.org/10.1016/j.imavis.2023.104784
Zhang H, Li S, Shi Y, Yang J (2019) Graph Fusion for Finger Multimodal Biometrics. IEEE Access 7:28607–28615. https://doi.org/10.1109/ACCESS.2019.2902133
Veeramachaneni K, Osadciw LA, Varshney PK (2005) “An adaptive multimodal biometric management algorithm,” in IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 35(3):344–356. https://doi.org/10.1109/TSMCC.2005.848191
Sultana M, Paul PP, Gavrilova ML (2018) Social Behavioral Information Fusion in Multimodal Biometrics. IEEE Trans Syst Man Cybernetics: Systems 48(12):2176–2187. https://doi.org/10.1109/TSMC.2017.2690321
Rida, I., Al-Maadeed, N., Al-Maadeed, S., & Bakshi, S. (2018). A comprehensive overview of feature representation for biometric recognition. In Multimedia Tools and Applications 79 7–8: 4867–4890. Springer Science and Business Media LLC. https://doi.org/10.1007/s11042-018-6808-5
Ghaderi A, Shahri AA, Larsson S (2022) A visualized hybrid intelligent model to delineate Swedish fine-grained soil layers using clay sensitivity. CATENA 214:106289
Abbaszadeh Shahri A, Shan C, Larsson S (2022) A novel approach to uncertainty quantification in groundwater table modeling by automated predictive deep learning. Nat Resour Res 31(3):1351–1373
Naik DL, Kiran R (2021) A novel sensitivity-based method for feature selection. Journal of Big Data 8:1–16
Asheghi R, Hosseini SA, Saneie M, Shahri AA (2020) Updating the neural network sediment load models using different sensitivity analysis methods: a regional application. J Hydroinf 22(3):562–577
Paul PP, Gavrilova ML, Alhajj R (2014) Decision Fusion for Multimodal Biometrics Using Social Network Analysis. IEEE Trans Syst Man Cybern: Syst 44(11):1522–1533. https://doi.org/10.1109/TSMC.2014.2331920
Bahrampour S, Nasrabadi NM, Ray A, Jenkins WK (2016) Multimodal Task-Driven Dictionary Learning for Image Classification. IEEE Trans Image Process 25(1):24–38. https://doi.org/10.1109/TIP.2015.2496275
Iula A, Micucci M (2022) Multimodal Biometric Recognition Based on 3D Ultrasound Palmprint-Hand Geometry Fusion. IEEE Access 10:7914–7925. https://doi.org/10.1109/ACCESS.2022.3143433
Jiang RM, Sadka AH, Crookes D (2010) “Multimodal Biometric Human Recognition for Perceptual Human-Computer Interaction,” in IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 40(6):676–681. https://doi.org/10.1109/TSMCC.2010.2050476
Rahman A et al (2021) Multimodal EEG and Keystroke Dynamics Based Biometric System Using Machine Learning Algorithms. IEEE Access 9:94625–94643. https://doi.org/10.1109/ACCESS.2021.3092840
Fox NA, Gross R, Cohn JF, Reilly RB (2007) Robust Biometric Person Identification Using Automatic Classifier Fusion of Speech, Mouth, and Face Experts. IEEE Trans Multimedia 9(4):701–714. https://doi.org/10.1109/TMM.2007.893339
Xin Y et al (2018) Multimodal Feature-Level Fusion for Biometrics Identification System on IoMT Platform. IEEE Access 6:21418–21426. https://doi.org/10.1109/ACCESS.2018.2815540
Toh KA, Yau WY (2004) "Combination of hyperbolic functions for multimodal biometrics data fusion. IEEE Trans Syst Man Cybern Part B 34(2):1196–1209. https://doi.org/10.1109/TSMCB.2003.821868
Talreja V, Valenti MC, Nasrabadi NM (2021) Deep Hashing for Secure Multimodal Biometrics. IEEE Trans Inf Forensics Secur 16:1306–1321. https://doi.org/10.1109/TIFS.2020.3033189
Li J, Hong D, Gao L, Yao J, Zheng K, Zhang B, Chanussot J (2022) Deep learning in multimodal remote sensing data fusion: A comprehensive review. Int J Appl Earth Obs Geoinformation (Vol. 112, p. 102926). Elsevier BV. https://doi.org/10.1016/j.jag.2022.102926
Wang Y, Shi D, Zhou W (2022) Convolutional Neural Network Approach Based on Multimodal Biometric System with Fusion of Face and Finger Vein Features. Sensors (Basel) 22(16):6039. https://doi.org/10.3390/s22166039
Gavrilova M, Luchak I, Sudhakar T, Tumpa SN (2022) Artificial Intelligence in Biometrics: Uncovering Intricacies of Human Body and Mind. In Learning and Analytics in Intelligent Systems (pp. 123–169). Springer International Publishing. https://doi.org/10.1007/978-3-030-93052-3_7
Funding
This research does not receive any kind of funding in any form.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
This research does not involve any animal or human participation.
Conflict of interest
The authors report no conflict of Interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Byeon, H., Raina, V., Sandhu, M. et al. Artificial intelligence-Enabled deep learning model for multimodal biometric fusion. Multimed Tools Appl 83, 80105–80128 (2024). https://doi.org/10.1007/s11042-024-18509-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-024-18509-0