Abstract
Cross-modality face recognition aims to match facial images across different modalities. This task becomes very challenging when one of the modalities is the facial caricature, which enhances instinctive facial features through extreme distortions and exaggerations with diverse styles by artists. In this paper, we develop a novel modality interference decoupling and representation alignment (MIR) method for visual-caricature face recognition. Our MIR method consists of a backbone network, an identity-interference orthogonal decoupling (IIOD) module, and a modality feature alignment (MFA) module. The IIOD module adopts a three-branch structure to decouple the deep semantic features extracted by the backbone network into identity features and modality features. In IIOD, we design an identity subspace alignment (ISA) module to align the identity features from different branches. Moreover, we design the MFA module to perform feature alignment between the modality feature from the IIOD module and that from the pre-trained modality interference information encoder (MIE) via adversarial learning, extracting the modality-specific information. Based on the above designs, we can effectively alleviate the interference of modality differences and style differences, improving the final performance. Extensive experimental results on multiple datasets show that our proposed method outperforms several state-of-the-art cross-modality face recognition methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dai, L., et al.: Gated fusion of discriminant features for caricature recognition. In: Proceedings of the International Conference on Intelligent Science and Big Data Engineering, pp. 563–573 (2019)
Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S.: RetinaFace: single-shot multi-level face localisation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5203–5212 (2020)
Fu, C., Wu, X., Hu, Y., Huang, H., He, R.: DVG-Face: dual variational generation for heterogeneous face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(6), 2938–2952 (2021)
Garg, J., Peri, S.V., Tolani, H., Krishnan, N.C.: Deep cross modal learning for caricature verification and identification (CaVINet). In: Proceedings of the ACM International Conference on Multimedia, pp. 1101–1109 (2018)
He, R., Wu, X., Sun, Z., Tan, T.: Learning invariant deep representation for NIR-VIS face recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2000–2006 (2017)
He, R., Wu, X., Sun, Z., Tan, T.: Wasserstein CNN: learning invariant features for NIR-VIS face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1761–1773 (2018)
Hu, W., Hu, H.: Orthogonal modality disentanglement and representation alignment network for NIR-VIS face recognition. IEEE Trans. Circ. Syst. Video Technol. 32(6), 3630–3643 (2021)
Hu, W., Hu, H.: Domain-private factor detachment network for NIR-VIS face recognition. IEEE Trans. Inf. Forensics Secur. 17, 1435–1449 (2022)
Huo, J., Gao, Y., Shi, Y., Yin, H.: Variation robust cross-modal metric learning for caricature recognition. In: Proceedings of the on Thematic Workshops of ACM Multimedia, pp. 340–348 (2017)
Huo, J., Li, W., Shi, Y., Gao, Y., Yin, H.: WebCaricature: a benchmark for caricature recognition. arXiv preprint arXiv:1703.03230 (2017)
Li, W., Huo, J., Shi, Y., Gao, Y., Wang, L., Luo, J.: A joint local and global deep metric learning method for caricature recognition. In: Proceedings of the Asian Conference on Computer Vision, pp. 240–256 (2019)
Liu, D., Gao, X., Peng, C., Wang, N., Li, J.: Heterogeneous face interpretable disentangled representation for joint face recognition and synthesis. IEEE Trans. Neural Netw. Learn. Syst. 33(10), 5611–5625 (2021)
Ming, Z., Burie, J.C., Muzzamil Luqman, M.: Dynamic deep multi-task learning for caricature-visual face cecognition. In: Proceedings of the International Conference on Document Analysis and Recognition Workshops, pp. 92–97 (2019)
Mishra, A.: DHFML: deep heterogeneous feature metric learning for matching photograph and caricature pairs. Int. J. Multimedia Inf. Retr. 8(3), 135–142 (2019)
Mishra, A., Rai, S.N., Mishra, A., Jawahar, C.: IIIT-CFW: a benchmark database of cartoon faces in the wild. In: Proceedings of the European Conference on Computer Vision, pp. 35–47 (2016)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4278–4284 (2017)
Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: Proceedings of the European Conference on Computer Vision, pp. 499–515 (2016)
Wu, X., Huang, H., Patel, V.M., He, R., Sun, Z.: Disentangled variational representation for heterogeneous face recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9005–9012 (2019)
Wu, X., Song, L., He, R., Tan, T.: Coupled deep learning for heterogeneous face recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1679–1686 (2018)
Yang, Z., Liang, J., Fu, C., Luo, M., Zhang, X.Y.: Heterogeneous face recognition via face synthesis with identity-attribute disentanglement. IEEE Trans. Inf. Forensics Secur. 17, 1344–1358 (2022)
Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv preprint arXiv:1411.7923 (2014)
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grants 62372388, 62071404 and by the Natural Science Foundation of Fujian Province under Grant 2020J01001.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, Y. et al. (2024). Modality Interference Decoupling and Representation Alignment for Caricature-Visual Face Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14425. Springer, Singapore. https://doi.org/10.1007/978-981-99-8429-9_24
Download citation
DOI: https://doi.org/10.1007/978-981-99-8429-9_24
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8428-2
Online ISBN: 978-981-99-8429-9
eBook Packages: Computer ScienceComputer Science (R0)