Skip to main content
Log in

Personalized Convolution for Face Recognition

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Face recognition has been significantly advanced by deep learning based methods. In all face recognition methods based on convolutional neural network (CNN), the convolutional kernels for feature extraction are fixed regardless of the input face once the training stage is finished. By contrast, we humans are usually impressed by some unique characteristics of different persons, such as one’s blue eyes while another one’s crooked nose, or even someone’s naevus at specific location. Inspired by this observation, we propose a personalized convolution method which aims to extract special distinguishing characteristics of each person for more accurate face recognition. Specifically, given a face, we adaptively generate a set of kernels for him/her, named by us ordinary kernel, which is further analytically decomposed into two orthogonal components, i.e., the commonality component and the specialty component. The former characterizes the commonality among subjects which is optimized on a reference set. The latter is the residual part by filtering out the commonality component from the ordinary kernel, so as to capture those special characteristics, named by us personalized kernel. The CNNs with personalized kernels for convolution can highlight those specialty of a person’s distinguishing characteristics while suppress his/her commonality with others, leading to better distinguishing of different faces. Additionally, as a by-product, the reference set also facilitates the adaptation of our method to different scenarios by simply selecting faces of a particular population. Extensive experiments on the challenging LFW, IJB-A and IJB-C datasets validate that our proposed personalized convolution achieves significant improvement over the conventional CNN, and also other existing methods for face recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • AbdAlmageed, W., Wu, Y., Rawls, S., Harel, S., Hassner, T., Masi, I., Choi, J., Lekust, J., Kim, J., Natarajan, P., et al. (2016). Face recognition using deep multi-pose representations. In Winter conference on applications of computer vision (WACV).

  • Ahonen, T., Hadid, A., & Pietikainen, M. (2006). Face description with local binary patterns: Application to face recognition. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2037–2041.

  • Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 711–720

  • Bertinetto, L., Henriques, J. F., Valmadre, J., Torr, P., & Vedaldi, A. (2016). Learning feed-forward one-shot learners. In Advances in neural information processing systems (NeurIPS).

  • Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) Vggface2: A dataset for recognising faces across pose and age. In International conference on automatic face and gesture recognition (FG).

  • Cao, Z., Yin, Q., Tang, X., & Sun, J. (2010). Face recognition with learning-based descriptor. In Conference on computer vision and pattern recognition (CVPR).

  • Chen, D., Cao, X., Wang, L., Wen, F., & Sun J. (2012). Bayesian face revisited: A joint formulation. In European Conference on Compute Vision (ECCV).

  • Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2017). StyleBank: An explicit representation for neural image style transfer. In Conference on computer vision and pattern recognition (CVPR).

  • Chen, J. C., Patel, V. M., Chellappa, R. (2016). Unconstrained face verification using deep CNN features. In Winter conference on applications of computer vision (WACV).

  • Chen, Y., Chen, Y., Wang, X., Tang, X. (2014). Deep learning face representation by joint identification-verification. In Advances in neural information processing systems (NeurIPS).

  • Dayong, W., Charles, O., Jain, A. K. (2017). Face search at scale. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 1122–1136.

  • Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2018). Arcface: Additive angular margin loss for deep face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Deng, W., Hu, J., & Guo, J. (2012). Extended src: Undersampled face recognition via intraclass variant dictionary. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 1864–1870.

  • Ding, C., & Tao, D. (2015). Robust face recognition via multimodal deep face representation. Transactions on Multimedia (TMM) 2049–2058.

  • Duan, Y., Lu, J., Feng, J., & Zhou, J. (2018). Context-aware local binary feature learning for face recognition. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 1139–1153.

  • Duan, Y., Lu, J., & Zhou, J. (2019). Uniformface: Learning deep equidistributed representation for face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Gong, S., Liu, X., & Jain, A. K. (2020). Jointly de-biasing face recognition and demographic attribute estimation. In European Conference on Computer Vision (ECCV).

  • Han, C., Shan, S., Kan, M., Wu, S., & Chen, X. (2018). Face recognition with contrastive convolution. In European conference on computer vision (ECCV).

  • He, X., Yan, S., Hu, Y., Niyogi, P., & Zhang, H. J. (2005). Face recognition using laplacianfaces. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 328–340.

  • Huang, G. B., & Learned-Miller, E. (2014). Labeled faces in the wild: Updates and new reporting procedures. In Department of Computer Science, Univ. Massachusetts Amherst, Tech. Rep.

  • Huang, G. B., Mattar, M., Berg, T., & Learned-Miller, E. (2008). Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on faces in ’Real-Life’ Images: detection, alignment, and recognition.

  • Huang, Y., Shen, P., Tai, Y., Li, S., Liu, X., Li, J., Huang, F., & Ji, R. (2020a). Improving face recognition from hard samples via distribution distillation loss. In European Conference on Computer Vision (ECCV).

  • Huang, Y., Wang, Y., Tai, Y., Liu, X., Shen, P., Li, S., Li, J., & Huang, F. (2020b). Curricularface: Adaptive curriculum learning loss for deep face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Jia, X., De Brabandere, B., Tuytelaars, T., & Gool, L. V. (2016). Dynamic filter networks. In Advances in neural information processing systems (NeurIPS).

  • Jian, Z., Yu, C., Yi, C., Yang, Y., Fang, Z., Jianshu, L., Hengzhu, L., Shuicheng, Y., & Jiashi, F. (2019). Look across elapse: Disentangled representation learning and photorealistic cross-age face synthesis for age-invariant face recognition. In Conference on artificial intelligence (AAAI).

  • Kang, B. N., Kim,Y., & Kim, D. (2018). Pairwise relational networks for face recognition. In European Conference on Computer Vision (ECCV).

  • Kang, B. N., Kim, Y., Jun, B., Kim, D. (2019). Attentional feature-pair relation networks for accurate face recognition. In International conference on computer vision (ICCV).

  • Kang, D., Dhar, D., & Chan, A. (2017). Incorporating side information by adaptive convolution. In Advances in neural information processing systems (NeurIPS).

  • Kim, Y., Park, W., Roh, M. C., Shin, J. (2020a). Groupface: Learning latent groups and constructing group-based representations for face recognition. In Conference on Computer Vision and Pattern Recognition (CVPR).

  • Kim, Y., Park, W., & Shin, J. (2020b). Broadface: Looking at tens of thousands of people at once for face recognition. In European Conference on Computer Vision (ECCV).

  • Klare, B. F., Klein, B., Taborsky, E., Blanton, A. (2015). Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In Conference on computer vision and pattern recognition (CVPR).

  • Klein, B., Wolf, L., & Afek, Y. (2015). A dynamic convolutional layer for short range weather prediction. In Conference on computer vision and pattern recognition (CVPR).

  • Lei, Z., Pietikäinen, M., & Li, S. Z. (2014). Learning discriminant face descriptor. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 289–302.

  • Liao, S., & Shao, L. (2019). Interpretable and generalizable deep image matching with adaptive convolutions. Computing Research Repository (CoRR).

  • Liu, C., & Wechsler, H. (2002). Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. Transactions on Image processing (TIP) 467–476.

  • Liu, W., Wen, Y., Yu, Z., & Yang, M. (2016). Large-margin softmax loss for convolutional neural networks. In International conference on machine learning (ICML).

  • Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., & Song, L. (2017). Sphereface: Deep hypersphere embedding for face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Luan, T., Yin, X., & Liu, X. (2017). Disentangled representation learning GAN for pose-invariant face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Masi, I., Rawls, S., Medioni, G., & Natarajan, P. (2016a). Pose-aware face recognition in the wild. In Conference on computer vision and pattern recognition (CVPR).

  • Masi, I., Tran, A. T., Leksut, J. T., Hassner, T., & Medioni, G. G. (2016b). Do we really need to collect millions of faces for effective face recognition? In European conference on computer vision (ECCV).

  • Maze, B., Adams, J. C., Duncan, J. A., Kalka, N. D., Miller, T., Otto, C., Jain, A. K., Niggel, W. T., Anderson, J., Cheney, J., & Grother, P. (2018). Iarpa janus benchmark - c: Face dataset and protocol. In International conference on biometrics (ICB) (pp. 158–165).

  • Moghaddam, B., Wahid, W., & Pentland, A. (1998). Beyond eigenfaces: Probabilistic matching for face recognition. In International conference on automatic face and gesture recognition (FG).

  • Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015). Deep face recognition. In British machine vision conference (BMVC).

  • Sankaranarayanan, S., Alavi, A., & Chellappa, R. (2016). Triplet similarity embedding for face verification. Preprint arXiv:160203418

  • Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A unified embedding for face recognition and clustering. In Conference on computer vision and pattern recognition (CVPR).

  • Shen, Y., Luo, P., Yan, J., Wang, X., & Tang, X. (2018). Faceid-gan: Learning a symmetry three-player gan for identity-preserving face synthesis. In Conference on computer vision and pattern recognition (CVPR).

  • Song, L., Gong, D., Li, Z., Liu, C., & Liu, W. (2019). Occlusion robust face recognition based on mask learning with pairwise differential siamese network. In International conference on computer vision (ICCV).

  • Sun, Y., Wang, X., & Tang, X. (2014). Deep learning face representation from predicting 10,000 classes. In Conference on computer vision and pattern recognition (CVPR).

  • Sun, Y., Liang, D., Wang, X., & Tang, X. (2015a). Deepid3: Face recognition with very deep neural networks. Preprint arXiv:150200873

  • Sun, Y., Wang, X., & Tang, X. (2015b). Deeply learned face representations are sparse, selective, and robust. In Conference on computer vision and pattern recognition (CVPR).

  • Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014). DeepFace: Closing the gap to human-level performance in face verification. In Conference on computer vision and pattern recognition (CVPR).

  • Tan, X., & Triggs, B. (2010). Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Transactions on Image Processing (TIP).

  • Tian, Z., Shen, C., & Chen, H. (2020). Conditional convolutions for instance segmentation. In European Conference on Computer Vision (ECCV).

  • Tuan Tran, A., Hassner, T., Masi, I., & Medioni, G. (2017). Regressing robust and discriminative 3d morphable models with a very deep neural network. In Conference on computer vision and pattern recognition (CVPR).

  • Turk, M., & Pentland, A. (1991). Eigenfaces for recognition. Journal of Cognitive Neuroscience 71–86.

  • Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., & Liu, W. (2018). Cosface: Large margin cosine loss for deep face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Wang, H., Gong, D., Li, Z., & Liu, W. (2019). Decorrelated adversarial learning for age-invariant face recognition. In Conference on computer vision and pattern recognition (CVPR).

  • Wang, M., & Deng, W. (2018). Deep face recognition: A survey. Computing Research Repository (CoRR).

  • Wenchao., Z, Shiguang., S, Wen., G, Xilin., C, & Hongming., Z. (2005) Local gabor binary pattern histogram sequence (lgbphs): a novel non-statistical model for face representation and recognition. In International conference on computer vision (ICCV).

  • Whitelam, C., Taborsky, E., Blanton, A., Maze, B., Adams, J., Miller, T., Kalka, N., Jain, A. K., Duncan, J. A., & Allen, K., et al. (2017). Iarpa janus benchmark-b face dataset. In Conference on computer vision and pattern recognition workshops (CVPRW).

  • Wolf, L., Hassner, T., & Maoz, I. (2011). Face recognition in unconstrained videos with matched background similarity. In Conference on computer vision and pattern recognition (CVPR).

  • Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 210–227.

  • Wu, W., Kan, M., Liu, X., Yang, Y., Shan, S., & Chen, X. (2017). Recursive spatial transformer (ReST) for alignment-free face recognition. In International conference on computer vision (ICCV).

  • Xie, S., Shan, S., Chen, X., & Chen, J. (2010). Fusing local patterns of gabor magnitude and phase for face recognition. Transactions on Image Processing (TIP) 1349–1361.

  • Xie, W., Shen, L., & Zisserman, A. (2018). Comparator networks. In European conference on computer vision (ECCV).

  • Yan, S., Xu, D., Zhang, B., & Zhang, H. J. (2005). Graph embedding: A general framework for dimensionality reduction. In Conference on computer vision and pattern recognition (CVPR).

  • Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014). Learning face representation from scratch. Preprint arXiv:14117923

  • Yin, X., Yu, X., Sohn, K., Liu, X., & Chandraker, M. (2017). Towards large-pose face frontalization in the wild. In International conference on computer vision (ICCV).

  • Zhang, B., Shan, S., Chen, X., & Gao, W. (2007) Histogram of gabor phase patterns (HGPP): A novel object representation approach for face recognition. Transactions on Image Processing (TIP) 57–68.

  • Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. Signal Processing Letters 1499–1503.

  • Zhang, L., Yang, M., & Feng, X. (2011). Sparse representation or collaborative representation: Which helps face recognition? In International conference on computer vision (ICCV).

  • Zhang, R., Tang, S., Zhang, Y., Li, J., & Yan, S. (2017). Scale-adaptive convolutions for scene parsing. In International conference on computer vision (ICCV).

  • Zhao, K., Xu, J., & Cheng, M. M. (2019). Regularface: Deep face recognition via exclusive regularization. In Conference on computer vision and pattern recognition (CVPR).

Download references

Acknowledgements

This work was partially supported the National Key Research and Development Program of China (No. 2017YFA0700800), the Natural Science Foundation of China (No. 61772496), and the Beijing Nova Program (Z191100001119123).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Meina Kan.

Additional information

Communicated by Bumsub Ham.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, C., Shan, S., Kan, M. et al. Personalized Convolution for Face Recognition. Int J Comput Vis 130, 344–362 (2022). https://doi.org/10.1007/s11263-021-01536-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01536-x

Keywords

Navigation