Skip to main content

Advertisement

Springer Nature Link
Log in

A deep feature-level fusion model for masked face identity recommendation system

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

The widespread occurrences of airborne outbreaks (e.g., COVID-19) and pollution (e.g., PM2.5) have urged people in the affected regions to protect themselves by wearing face masks. In certain areas, wearing masks amidst such health-endangering times is even enforced by law. While most people wear masks to guard themselves against airborne substances, some exploit such excuses and use face masks to conceal their identity for criminal purposes such as shoplifting, robbery, drug transport, and assault. While automatic face recognition models have been proposed, most of these models aim to identify clear, unobstructed faces for authentication purposes and cannot effectively handle cases where masks cover most facial areas. To mitigate such a problem, this paper proposes a deep-learning-based feature-fusion framework, FIREC, that combines additional demographic-estimated features such as age, gender, and race into the underlying facial representation to compensate for the information lost due to mask obstruction. Given an image of a masked face, our system recommends a ranked list of potential identities of the person behind the mask. Empirical results show that the best configuration of our proposed framework can recognize bare faces and masked faces with the accuracy of 99.34% and 97.65% in terms of Hit@10, respectively. The proposed framework could greatly benefit high-recall facial identity recognition applications such as identifying potential suspects from CCTV or passers-by’s cameras, especially during crisis times when people commonly cover their faces with protective masks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Code availability

The source code is made available for research purposes at https://github.com/Zenonist/FIREC.

Notes

  1. https://github.com/Zenonist/FIREC.

  2. http://www.robots.ox.ac.uk/~vgg/data/vgg_face/.

  3. http://www.robots.ox.ac.uk/~vgg/data/vgg_face.

  4. https://keras.io/.

  5. https://www.tensorflow.org/.

References

  • Ahmed T, Das P, Ali MF, Mahmud M-F (2020) A comparative study on convolutional neural network based face recognition. In: 2020 11th international conference on computing, communication and networking technologies (ICCCNT). IEEE, pp 1–5

  • Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041. https://doi.org/10.1109/tpami.2006.244

    Article  Google Scholar 

  • Ahonen T, Rahtu E, Ojansivu V, Heikkila J (2008) Recognition of blurred faces using local phase quantization. In: 2008 19th International conference on pattern recognition. https://doi.org/10.1109/icpr.2008.4761847

  • Anwar A, Raychowdhury A (2020) Masked face recognition for secure authentication. arXiv preprint. arXiv:2008.11104

  • Arigbabu OA, Ahmad SMS, Adnan WAW, Mahmood S (2015) Soft biometrics: gender recognition from unconstrained face images using local feature descriptor. J Inf Commun Technol. https://doi.org/10.32890/jict2015.14.0.8159

    Article  Google Scholar 

  • Aswal V, Tupe O, Shaikh S, Charniya NN (2020) Single camera masked face identification. In: 2020 19th IEEE international conference on machine learning and applications (ICMLA). https://doi.org/10.1109/icmla51294.2020.00018

  • Beveridge JR, Zhang H, Draper BA, Flynn PJ, Feng Z, Huber P, Kittler J, Huang Z, Li S, Li Y et al (2015) Report on the FG 2015 video person recognition evaluation. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG). https://doi.org/10.1109/fg.2015.7163156

  • Bonnen K, Klare BF, Jain AK (2013) Component-based representation in automated face recognition. IEEE Trans Inf Forensics Secur 8(1):239–253. https://doi.org/10.1109/tifs.2012.2226580

    Article  Google Scholar 

  • Chang X, Wu J, Yang T, Feng G (2020) Deepfake face image detection based on improved VGG convolutional neural network. In: 2020 39th Chinese control conference (CCC). https://doi.org/10.23919/ccc50068.2020.9189596

  • Chen J-C, Ranjan R, Patel VM, Castillo CD, Chellappa R (2018) Unconstrained face identification and verification using deep convolutional features. In: Deep learning in biometrics, pp 33–64. https://doi.org/10.1201/b22524-2

  • Chen Y-N, Thaipisutikul T, Han C-C, Liu T-J, Fan K-C (2021) Feature line embedding based on support vector machine for hyperspectral image classification. Remote Sens 13(1):130. https://doi.org/10.3390/rs13010130

    Article  Google Scholar 

  • Chu W, Ying Z, Xia X (2013) Facial expression recognition based on binarized statistical image features. In: 2013 Ninth international conference on natural computation (ICNC). IEEE, pp 328–332

  • Dantas AJ, Jesus LD, Ramos ACB, Hokama P, Mora-Camino F, Katarya R, Verma OP, Grupta PK, Singh G, Ouahada K (2021) Using UAV, IoMT and AI for monitoring and supplying of COVID-19 patients. In: ITNG 2021 18th international conference on information technology-new generations. Springer, Berlin, pp 383–386

  • Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531. https://doi.org/10.1109/tpami.2015.2462338

    Article  Google Scholar 

  • Dosaj A, Satapathy SC, Soundrapandiyan R, Kaur M, Hannoon N (2018) An efficient cloud based face recognition system for e-health secured login using steerable pyramid transform and local directional pattern. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-018-1115-6

    Article  Google Scholar 

  • Enkhbat A, Shih TK, Thaipisutikul T, Hakim NL, Aditya W (2020) Handkey: an efficient hand typing recognition using CNN for virtual keyboard. In: 2020—5th International conference on information technology (InCIT). https://doi.org/10.1109/incit50588.2020.9310783

  • Guo C (2017) Enhancing face identification using local binary patterns and k-nearest neighbors. J Imaging 3(3):37. https://doi.org/10.3390/jimaging3030037

    Article  Google Scholar 

  • Gupta A, Katarya R (2021a) A novel LDA-based framework to forecast COVID-19 trends. Available at SSRN 3833706

  • Gupta A, Katarya R (2021b) PAN-LDA: a latent Dirichlet allocation based novel feature extraction model for COVID-19 data using machine learning. Comput Biol Med 138:104920

    Article  Google Scholar 

  • Gupta G, Katarya R (2021c) EnPSO: an AutoML technique for generating ensemble recommender system. Arab J Sci Eng 46(9):8677–8695

    Article  Google Scholar 

  • Gupta G, Katarya R (2021d) A study of deep reinforcement learning based recommender systems. In: 2021 2nd international conference on secure cyber computing and communications (ICSCCC). IEEE, pp 218–220

  • Gupta A, Gupta S, Katarya R et al (2021) InstaCovNet-19: a deep learning classification model for the detection of COVID-19 patients using chest X-ray. Appl Soft Comput 99:106859

    Article  Google Scholar 

  • Jose JP, Poornima P, Kumar KM (2012) A novel method for color face recognition using KNN classifier. In: 2012 International conference on computing, communication and applications. IEEE, pp 1–3

  • Kar A, Pramanik S, Chakraborty A, Bhattacharjee D, Ho ESL, Shum HPH (2020) LMZMPM: local modified Zernike moment per-unit mass for robust human face recognition. IEEE Trans Inf Forensics Secur 16:495–509

    Article  Google Scholar 

  • Karaaba M, Surinta O, Schomaker L, Wiering MA (2015) Robust face recognition by computing distances from multiple histograms of oriented gradients. In: 2015 IEEE symposium series on computational intelligence. IEEE, pp 203–209

  • Katarya Rahul, Arora Yamini (2020) Capsmf: a novel product recommender system using deep learning based text analysis model. Multimedia Tools and Applications 79(47):35927–35948

    Article  Google Scholar 

  • Katarya R, Saini R (2022) Enhancing the wine tasting experience using greedy clustering wine recommender system. Multimedia Tools Appl 81(1):807–840

    Article  Google Scholar 

  • Katarya R, Verma OP, Jain I (2013) User behaviour analysis in context-aware recommender system using hybrid filtering approach. In: 2013 4th International conference on computer and communication technology (ICCCT). IEEE, pp 222–227

  • Katarya R, Gupta A, Sachdeva S, Dhamija T, Gupta S, Gupta A, Kedia P, Rai V et al (2021) A review of various mathematical and deep learning based forecasting methods for COVID-19 pandemic. In: 2021 7th International conference on advanced computing and communication systems (ICACCS), vol 1. IEEE, pp 874–878

  • Kedia P, Katarya R et al (2021) CoVNet-19: a deep learning model for the detection and analysis of COVID-19 patients. Appl Soft Comput 104:107184

    Article  Google Scholar 

  • Kusakunniran W, Karnjanapreechakorn S, Siriapisith T, Borwarnginn P, Sutassananon K, Tongdee T, Saiviroonporn P (2021) COVID-19 detection and heatmap generation in chest x-ray images. J Med Imaging 8(S1):014001

    Article  Google Scholar 

  • Li X-X, Hao P, He L, Feng Y (2020) Image gradient orientations embedded structural error coding for face recognition with occlusion. J Ambient Intell Humaniz Comput 11(6):2349–2367

    Article  Google Scholar 

  • Li X, Du Z, Huang Y, Tan Z (2021) A deep translation (GAN) based change detection network for optical and SAR remote sensing images. ISPRS J Photogramm Remote Sens 179:14–34

    Article  Google Scholar 

  • Ma Z, Xiang Z (2015) Robust visual tracking via binocular multi-task multi-view joint sparse representation. In: 2015 SAI intelligent systems conference (IntelliSys). IEEE, pp 714–722

  • Naveen P, Sivakumar P (2021) Adaptive morphological and bilateral filtering with ensemble convolutional neural network for pose-invariant face recognition. J Ambient Intell Humaniz Comput 12(11):10023–10033

    Article  Google Scholar 

  • Qi G-J, Hua X-S, Rui Y, Tang J, Zhang H-J (2009) Two-dimensional multilabel active learning with an efficient online adaptation model for image classification. IEEE Trans Pattern Anal Mach Intell 31(10):1880–1897. https://doi.org/10.1109/tpami.2008.218

    Article  Google Scholar 

  • Ojansivu V, Heikkilä J (2008) Blur insensitive texture classification using local phase quantization. In: International conference on image and signal processing. Springer, Berlin, pp 236–243

  • Sarangi PP, Nayak DR, Panda M, Majhi B (2022) A feature-level fusion based improved multimodal biometric recognition system using ear and profile face. J Ambient Intell Human Comput 13:1867–1898. https://doi.org/10.1007/s12652-021-02952-0

    Article  Google Scholar 

  • Serengil SI, Ozpinar A (2020) Lightface: a hybrid deep face recognition framework. In: 2020 Innovations in intelligent systems and applications conference (ASYU). IEEE, pp 23–27. https://doi.org/10.1109/ASYU50717.2020.9259802

  • Simonyan K, Parkhi O, Vedaldi A, Zisserman A (2013) Fisher vector faces in the wild. In: Proceedings of the British machine vision conference 2013,. https://doi.org/10.5244/c.27.8

  • Sun J, Fu Y, Li S, He J, Xu C, Tan L (2018) Sequential human activity recognition based on deep convolutional network and extreme learning machine using wearable sensors. J Sens 1–10:2018. https://doi.org/10.1155/2018/8580959

    Article  Google Scholar 

  • Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701–1708

  • VenkateswarLal P, Nitta GR, Prasad A (2019) Ensemble of texture and shape descriptors using support vector machine classification for face recognition. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-019-01192-7

    Article  Google Scholar 

  • Vera-Rodriguez R, Blazquez M, Morales A, Gonzalez-Sosa E, Neves JC, Proença H (2019) FaceGenderID: exploiting gender information in DCNNs face recognition systems. In: 2019 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 2254–2260. https://doi.org/10.1109/CVPRW.2019.00278

  • Xu C, Tao D, Xu C (2014) Large-margin multi-viewinformation bottleneck. IEEE Trans Pattern Anal Mach Intell 36(8):1559–1572. https://doi.org/10.1109/tpami.2013.2296528

    Article  Google Scholar 

  • Yolcu G, Oztel I, Kazan S, Oz C, Bunyak F (2019) Deep learning-based face analysis system for monitoring customer interest. J Ambient Intell Humaniz Comput 11(1):237–248. https://doi.org/10.1007/s12652-019-01310-5

    Article  Google Scholar 

  • Zhu X, Liao S, Lei Z, Liu R, Li SZ (2007) Feature correlation filter for face recognition. In: International conference on biometrics. Springer, Berlin, pp 77–86

Download references

Acknowledgements

This research project is supported by Mahidol University (Grant no. MU-MiniRC02/2564). Suppawong Tuarob is the corresponding author.

Author information

Authors and Affiliations

  1. Faculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, Thailand

    Tipajin Thaipisutikul, Phonarnun Tatiyamaneekul & Suppawong Tuarob

  2. Department of Electrical Engineering, Yuan Ze University, Taoyuan, Taiwan

    Chih-Yang Lin

Authors
  1. Tipajin Thaipisutikul
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Phonarnun Tatiyamaneekul
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Chih-Yang Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Suppawong Tuarob
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Suppawong Tuarob.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thaipisutikul, T., Tatiyamaneekul, P., Lin, CY. et al. A deep feature-level fusion model for masked face identity recommendation system. J Ambient Intell Human Comput 14, 16011–16024 (2023). https://doi.org/10.1007/s12652-022-04380-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-04380-0

Keywords