research-article

Heterogeneous Face Recognition with Attention-guided Feature Disentangling

Authors:

Jianwei ZhangAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 4137 - 4145

https://doi.org/10.1145/3474085.3475546

Published: 17 October 2021 Publication History

Abstract

This paper proposes an attention-guided feature disentangling framework (AgFD) to eliminate the large cross-modality discrepancy for Heterogeneous Face Recognition (HFR). Existing HFR methods either focus only on extracting identity features or impose linear/no independence constraints on the decomposed components. Instead, our AgFD disentangles the facial representation and forces intrinsic independence between identity features and identity-irrelevant variations. To this end, an Attention-based Residual Decomposition Module (AbRDM) and an Adversarial Decorrelation Module (ADM) are presented. AbRDM provides hierarchical complementary feature disentanglement, while ADM is introduced for decorrelation learning. Extensive experiments on the challenging CASIA NIR-VIS 2.0 Database, Oulu-CASIA NIR&VIS Database, BUAA-VisNir Database, and IIIT-D Viewed Sketch Database demonstrate the generalization ability and competitive performance of the proposed method.

References

[1]

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In International Conference on Machine Learning. 531--540.

[2]

H.S. Bhatt, R. Singh S. Bharadwaj, and M. Vatsa. 2011. Memetic Approach for Matching Sketches with Digital Face Images. Technical report, IIITD-TR-2011-006 (2011).

[3]

Himanshu S Bhatt, Samarth Bharadwaj, Richa Singh, and Mayank Vatsa. 2010. On matching sketches with digital face images. In Fourth IEEE International Conference on Biometrics: Theory Applications and Systems (BTAS). 1--7.

[4]

Himanshu S Bhatt, Samarth Bharadwaj, Richa Singh, and Mayank Vatsa. 2012. Memetically optimized MCWLD for matching sketches with digital face images. IEEE Transactions on Information Forensics and Security, Vol. 7, 5 (2012), 1522--1535.

Digital Library

[5]

Bing Cao, Nannan Wang, Xinbo Gao, Jie Li, and Zhifeng Li. 2019. Multi-Margin based Decorrelation Learning for Heterogeneous Face Recognition. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. 680--686. https://doi.org/10.24963/ijcai.2019/96

Digital Library

[6]

Bing Cao, Nannan Wang, Jie Li, and Xinbo Gao. 2018. Data augmentation-based joint learning for heterogeneous face recognition. IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, 6 (2018), 1731--1743.

[7]

Jie Chen, Shiguang Shan, Chu He, Guoying Zhao, Matti Pietikainen, Xilin Chen, and Wen Gao. 2010. WLD: A robust local image descriptor. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, 9 (2010), 1705--1720.

Digital Library

[8]

Jie Chen, Dong Yi, Jimei Yang, Guoying Zhao, Stan Z Li, and Matti Pietikainen. 2009. Learning mappings for face synthesis from near infrared to visual light images. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 156--163.

[9]

Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems. 2610--2620.

[10]

Zhongying Deng, Xiaojiang Peng, and Yu Qiao. 2019. Residual compensation networks for heterogeneous face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8239--8246.

[11]

Boyan Duan, Chaoyou Fu, Yi Li, Xingguang Song, and Ran He. 2020. Cross-Spectral Face Hallucination via Disentangling Independent Factors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7930--7938.

[12]

Chanho Eom and Bumsub Ham. 2019. Learning Disentangled Representation for Robust Person Re-identification. In Advances in Neural Information Processing Systems. 5298--5309.

Digital Library

[13]

Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. 2019. Dual Variational Generation for Low Shot Heterogeneous Face Recognition. In Advances in Neural Information Processing Systems. 2674--2683.

Digital Library

[14]

Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropagation. In International Conference on Machine Learning. 1180--1189.

Digital Library

[15]

Weikuo Guo, Huaibo Huang, Xiangwei Kong, and Ran He. 2019. Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation. In Proceedings of the 27th ACM International Conference on Multimedia. 1712--1720.

Digital Library

[16]

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In European Conference on Computer Vision. 87--102.

[17]

Ran He, Jie Cao, Lingxiao Song, Zhenan Sun, and Tieniu Tan. 2019. Adversarial Cross-Spectral Face Completion for NIR-VIS Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42 (2019), 1025--1037.

[18]

Ran He, Yi Li, Xiang Wu, Lingxiao Song, Zhenhua Chai, and Xiaolin Wei. 2021. Coupled adversarial learning for semi-supervised heterogeneous face recognition. Pattern Recognition, Vol. 110 (2021), 107618.

[19]

Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. 2017. Learning invariant deep representation for nir-vis face recognition. In Thirty-First AAAI Conference on Artificial Intelligence.

Digital Library

[20]

Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. 2018. Wasserstein cnn: Learning invariant features for nir-vis face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, 7 (2018), 1761--1773.

[21]

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations.

[22]

R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019,New Orleans, LA, USA, May 6-9, 2019.

[23]

Weipeng Hu and Haifeng Hu. 2020. Dual Adversarial Disentanglement and Deep Representation Decorrelation for NIR-VIS Face Recognition. IEEE Transactions on Information Forensics and Security, Vol. 16 (2020), 70--85.

[24]

D Huang, J Sun, and Y Wang. 2012. The buaa-visnir face database instructions. School Comput. Sci. Eng., Beihang Univ., Beijing, China, Tech. Rep. IRIP-TR-12-FR-001 (2012).

[25]

Yangru Huang, Peixi Peng, Yi Jin, Yidong Li, Junliang Xing, and Shiming Ge. 2020. Domain Adaptive Attention Learning for Unsupervised Person Re-Identification. In AAAI. 11069--11076.

[26]

Seyed Mehdi Iranmanesh, Benjamin Riggan, Shuowen Hu, and Nasser M Nasrabadi. 2020. Coupled generative adversarial network for heterogeneous face recognition. Image and Vision Computing, Vol. 94 (2020), 103861.

[27]

Hyunjik Kim and Andriy Mnih. 2018. Disentangling by Factorising. In International Conference on Machine Learning. 2649--2658.

[28]

Justin B Kinney and Gurinder S Atwal. 2014. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences, Vol. 111, 9 (2014), 3354--3359.

[29]

Brendan Klare and Anil K Jain. 2010. Sketch to Photo Matching: A Feature-based Approach. In Biometric technology for human identification VII, Vol. 7667. 766702.

[30]

Brendan Klare, Zhifeng Li, and Anil K. Jain. 2011. Matching Forensic Sketches to Mug Shot Photos. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 33, 3 (2011), 639--646.

Digital Library

[31]

José Lezama, Qiang Qiu, and Guillermo Sapiro. 2017. Not afraid of the dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6628--6637.

[32]

Stan Li, Dong Yi, Zhen Lei, and Shengcai Liao. 2013. The casia nir-vis 2.0 face database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 348--353.

Digital Library

[33]

Wei Li, Xiatian Zhu, and Shaogang Gong. 2018. Harmonious attention network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2285--2294.

[34]

Decheng Liu, Xinbo Gao, Nannan Wang, Jie Li, and Chunlei Peng. 2020. Coupled Attribute Learning for Heterogeneous Face Recognition. IEEE Transactions on Neural Networks and Learning Systems, Vol. PP, 99 (2020), 1--14.

[35]

Xiaoxiang Liu, Lingxiao Song, Xiang Wu, and Tieniu Tan. 2016. Transferring deep representation for NIR-VIS heterogeneous face recognition. In 2016 International Conference on Biometrics (ICB). IEEE, 1--8.

[36]

Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. 2019. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems. 15578--15588.

Digital Library

[37]

Christopher Reale, Nasser M Nasrabadi, Heesung Kwon, and Rama Chellappa. 2016. Seeing the forest from the trees: A holistic approach to near-infrared heterogeneous face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 54--62.

[38]

Shreyas Saxena and Jakob Verbeek. 2016. Heterogeneous face recognition with CNNs. In European Conference on Computer Vision. 483--491.

[39]

Sadia Sharmin, Mohammad Shoyaib, Amin Ahsan Ali, Muhammad Asif Hossain Khan, and Oksam Chae. 2019. Simultaneous feature selection and discretization based on mutual information. Pattern Recognition, Vol. 91 (2019), 162--174.

Digital Library

[40]

Hailin Shi, Xiaobo Wang, Dong Yi, Zhen Lei, Xiangyu Zhu, and Stan Z Li. 2016. Cross-modality face recognition via heterogeneous joint bayesian. IEEE Signal Processing Letters, Vol. 24, 1 (2016), 81--85.

[41]

Lingxiao Song, Man Zhang, Xiang Wu, and Ran He. 2018. Adversarial discriminative heterogeneous face recognition. In Thirty-Second AAAI Conference on Artificial Intelligence.

[42]

Greg Ver Steeg. 2017. Unsupervised Learning via Total Correlation Explanation. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 5151--5155.

Digital Library

[43]

Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1415--1424.

[44]

Hao Wang, Dihong Gong, Zhifeng Li, and Wei Liu. 2019. Decorrelated adversarial learning for age-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3527--3536.

[45]

Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. 2018a. A light cnn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security, Vol. 13, 11 (2018), 2884--2896.

[46]

Xiang Wu, Huaibo Huang, Vishal M Patel, Ran He, and Zhenan Sun. 2019. Disentangled variational representation for heterogeneous face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9005--9012.

Digital Library

[47]

Xiang Wu, Lingxiao Song, Ran He, and Tieniu Tan. 2018b. Coupled deep learning for heterogeneous face recognition. In Thirty-Second AAAI Conference on Artificial Intelligence.

[48]

Shanmin Yang, Keren Fu, Xiao Yang, Ye Lin, Jianwei Zhang, and Cheng Peng. 2020. Learning Domain-Invariant Discriminative Features for Heterogeneous Face Recognition. IEEE Access, Vol. 8 (2020), 209790--209801.

[49]

Dong Yi, Zhen Lei, and Stan Z Li. 2015. Shared representation learning for heterogenous face recognition. In 2015 11th IEEE international Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. 1--7.

[50]

Wei Zhang, Xiaogang Wang, and Xiaoou Tang. 2011. Coupled Information-Theoretic Encoding for Face Photo-Sketch Recognition. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. 513--520.

Digital Library

Cited By

Peng CKong ZLiu DWang NGao X(2024)Disguised Heterogeneous Face Generation With Iterative-Adversarial Style UnificationIEEE Transactions on Multimedia10.1109/TMM.2023.331497726(3741-3753)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3314977
Hu WYang YHu H(2024)Pseudo Label Association and Prototype-Based Invariant Learning for Semi-Supervised NIR-VIS Face RecognitionIEEE Transactions on Image Processing10.1109/TIP.2024.336453033(1448-1463)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3364530
Yang YHu WHu H(2024)Unsupervised NIR-VIS Face Recognition via Homogeneous-to-Heterogeneous Learning and Residual-Invariant EnhancementIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.334617619(2112-2126)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2023.3346176
Show More Cited By

Index Terms

Heterogeneous Face Recognition with Attention-guided Feature Disentangling
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Matching
      2. Computer vision tasks
        Biometrics
  2. Machine learning
    1. Machine learning algorithms
      1. Feature selection
2. Security and privacy
  1. Security services
    1. Authentication
      1. Biometrics

Recommendations

Age-Invariant Face Recognition by Multi-Feature Fusionand Decomposition with Self-attention
Different from general face recognition, age-invariant face recognition (AIFR) aims at matching faces with a big age gap. Previous discriminative methods usually focus on decomposing facial feature into age-related and age-invariant components, which ...
Coupled Discriminative Feature Learning for Heterogeneous Face Recognition
This paper presents a coupled discriminative feature learning (CDFL) method for heterogeneous face recognition (HFR). Different from most existing HFR approaches which use hand-crafted feature descriptors for face representation, our CDFL directly learns ...
Knowledge Guided Representation Disentanglement for Face Recognition from Low Illumination Images
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Low illumination face recognition is challenging as details are lacking due to lighting conditions. Retinex theory points out that images can be divided into reflectance with color constancy and ambient illumination. Inspired by this, we propose a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

the Sichuan Provincial S&T Projects
the National Natural Science Foundation of China

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
247
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)3

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Peng CKong ZLiu DWang NGao X(2024)Disguised Heterogeneous Face Generation With Iterative-Adversarial Style UnificationIEEE Transactions on Multimedia10.1109/TMM.2023.331497726(3741-3753)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3314977
Hu WYang YHu H(2024)Pseudo Label Association and Prototype-Based Invariant Learning for Semi-Supervised NIR-VIS Face RecognitionIEEE Transactions on Image Processing10.1109/TIP.2024.336453033(1448-1463)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3364530
Yang YHu WHu H(2024)Unsupervised NIR-VIS Face Recognition via Homogeneous-to-Heterogeneous Learning and Residual-Invariant EnhancementIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.334617619(2112-2126)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2023.3346176
Zhang SWang LCui ZWang S(2024)A sketch recognition method based on bi-modal model using cooperative learning paradigmNeural Computing and Applications10.1007/s00521-024-09836-236:23(14275-14290)Online publication date: 6-May-2024
https://doi.org/10.1007/s00521-024-09836-2
Kuo CTsai YShuai HYeh YHuang CMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Towards Understanding Cross Resolution Feature Matching for Surveillance Face RecognitionProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548402(6706-6716)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548402

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten