Abstract
Small sample and pose variation are two critical technical problems in automatic face recognition (AFR). The combination of these two difficulties will seriously affect the performance of face recognition and restrict the wide application of AFR. To address this problem, we propose a new multi-feature fusion framework. First, we propose a pixel-level data augmentation algorithm based on manifold subspace partition, which constructs virtual samples in the original face image space to achieve training sample expansion and diversity enhancement. Then, we propose a feature-level data augmentation based on Gabor transformation, which can capture multi-level face features through multi-scale and multi-direction Gabor filters to realize the face expansion in feature space. To eliminate the data redundancy and interference information generated by Gabor feature augmentation, a Gabor feature encoding algorithm is proposed to construct the compressed Gabor feature vector. In addition, we propose a small scale adaptive deep CNN model, which is suitable for small sample datasets and can effectively extract nonlinear deep features of pose-varied faces. Finally, Gabor encoding features and nonlinear deep features are combined for small sample face recognition with pose variation. Experiment results based on two face datasets prove the effectiveness of the proposed multi-feature fusion framework.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig9_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig10_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-020-09076-1/MediaObjects/11042_2020_9076_Fig12_HTML.png)
Similar content being viewed by others
References
Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell 28(12):2037–2041
Bao-cai Y, Zhuang Z, Yan-feng S, Cheng-zhang W (2007) Pose variant face recognition based on 3D Morphable model. J Beijing Univ Techn 33(3):320–325
Bengio Y, Lamblin P, Popovici D et al (2007) Greedy layer-wise training of deep networks. Adv Neural Inf Proces Syst:153–160
Beom-Seok O, Toh K-A, Teoh ABJ et al (2018) An analytic Gabor feed forward network for single-sample and pose-invariant face recognition. IEEE Trans Image Process 27(6):2791–2805
Cao B, Wang N, Li J et al (2018) Data augmentation-based joint learning for heterogeneous face recognition. IEEE Trans Neural Networks Learning Syst:1–13
Ye Changiming, Jiang Jianguo, Zhan Shu, et al. 3D Facial Depth Map Recognition in Different Poses with Surface Contour Feature. PR & AI, 2013,26(2):219–224
Cheng D, GuangDa S, Gang LX et al (2004) Synthesis of Face Image with Pose Variations. Journal of Optoelectronics·Laser 15(12):1498–1451
Choe J, Park S, Kim K et al (2017) Face Generation for Low-Shot Learning Using Generative Adversarial Networks. IEEE Int Conf Computer Vision (ICCV):1940–1948
Daugman J (1985) Uncertainty relation for resolution in space, spatial, frequency, and orientation optimized by two-dimensional visual cortical filters. J Opt Soc Am 2(7):1160–1169
De Marsico M, Nappi M, Riccio D et al (2013) Robust face recognition for uncontrolled pose and illumination changes. IEEE Trans Syst Man, Cybernetics: Syst 43(1):149–163
Fan Deng-Ping, Zhang Shengchuan, Wu Yu-Huan et al. Scoot: A Perceptual Metric for Facial Sketches. 2019 IEEE Int Conference on Computer Vision(ICCV2019),2019:1–11.
Ding C, Tao D (2015) Robust face recognition via multimodal deep face representation. IEEE Trans Multimedia 17(11):2049–2058
GuoFeng Z, GuiXia F, HaiTao L et al (2015) A survey of multi-pose face recognition. PR&AI 28(7):613–625
Guo-feng ZOU, Gui-xia FU, Ke-jun WANG, Ming-liang GAO, Jin SHEN (2017) Construction method of adaptive deep convolutional neural network model. J Beijing Univ Posts Telecomm 40(4):98–103
Huajie C, Wei W (2007) Multi pose face recognition based on correlative sub region mapping. Journal of Image and Graphics 12(7):1254–1260
KaiQi H, XiaoTang C, YunFeng K et al (2015) Intelligent Visual Surveillance: A Review. Chinese Journal Of Computer 38(6):1093–1115
Ke-Jun W, Guo-Feng Z (2013) A Sub-Pattern Gabor Features Fusion Method for Single Sample Face Recognition. Pattern Recog Artificial Intelligent 26(1):50–56
LeCun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Leng B, Yu K, Qin J (2017) Data augmentation for unbalanced face recognition training sets. Neurocomputing 235:10–14
Li L, Ge H, Tong Y, Zhang Y (2018) Face recognition using Gabor-based feature extraction and feature space transformation fusion method for single image per person problem. Neural Process Letter 47:1197–1217
Lian SF, Liu YH, Li LC (2014) Face recognition under unconstrained based on LBP and deep learning. J Commun 35(6):154–160
W Q Liang, G G Wang, J H Lai, et al. M2M-GAN: Many-to-Many Generative Adversarial Transfer Learning for Person Re-Identification.2018. https://arxiv.org/abs/1811.03768.
Z. X. Lin, W. J. Yang, C. C. Ho et al. Fast Vertical-Pose- Invariant Face Recognition Module for Intelligent Robot Guard. The 4th International Conference on Autonomous Robots and Agents, 2009:613–617.
Ziwei Liu,Ping Luo,Xiaogang Wang, et al. Deep Learning Face Attributes in the Wild, 2015 IEEE International Conference on Computer Vision: 3730–3738
J. Liu, Y. Deng, C. Huang, Targeting Ultimate Accuracy: Face Recognition via Deep Embedding, arXiv preprint arxiv:hepth/1506.07310.
Lowe DG (1999) Object recognition from local scale-invariant features, in: proceedings of the IEEE international conference on computer vision. Vol. 2:1150–1157
Luan S, Chen C, Zhang B et al (2018) Gabor convolutional networks. IEEE Trans Image Process 27(9):4357–4366
Lv J-J, Shao X-H, Huang J-S et al (2017) Data augmentation for face recognition. Neurocomputing 230:184–196
J M Lv, W H Chen, Q Li, et al. Unsupervised Cross-dataset Person Re-identification by Transfer Learning of Spatial-Temporal Patterns. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 7948–7956.
Iacopo Masi, Stephen Rawls, Gérard Medioni, Prem Natarajan. Pose-Aware Face Recognition in the Wild, 2016 .IEEE Int Conf Comp VisionPattern Recog:4838–4846
Masi I, Chang F-J, Choi J et al (2019) Learning pose-aware models for pose-invariant face recognition in the wild. EEE Trans Pattern Anal Mach Intel 41(2):379–393
Moon H-M, Seo CH, Pan SB (2017) A face recognition system based on convolution neural network using multiple distance face. Soft Comput 21(17):4995–5002
Ng A. Sparse autoencoder . CS294A Lecture notes, 2011, 72(2011): 1–19.
Guang Yu Nie, Ming Ming Cheng, Yun Liu, et al. Multi-Level Context Ultra-Aggregation for Stereo Matching. 2019 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2019)2019:3283–3291.
Nikolay P (1995) Biologically motivated computationally intensive approaches to image pattern recognition. Futur Gener Comput Syst 11(4/5):451–465
Nikolay P (1995) Biologically motivated computationally intensive approaches to image pattern recognition. Futur Gener Comput Syst 11(4/5):451–465
Yongri Piao, Wei Ji, Jingjing Li, et al. Depth-induced Multi-scale Recurrent Attention Network for Saliency Detection. 2019 IEEE International Conference on Computer Vision(ICCV2019),2019:7254–7263.
Hongwei Qin, Junjie Yan, Xiu Li, et al. Joint Training of Cascaded CNN for Face Detection. 2016 IEEE International Conference on Computer Vision and Pattern Recognition:3456–3465.
Sharma A, Al Haj M, Choi J et al (2012) Robust pose invariant face recognition using coupled latent space discriminant analysis. Comput Vis Image Underst 116:1095–1110
Y Sun, X Wang, X Tang. Hybrid deep learning for face verification. Proc. IEEE Int Conference Comp Vision (ICCV 2013), 2013:1489–1496.
Y. Taigman, M. Yang, M. Ranzato, L. Wolf, Deepface: closing the gap to humanlevel performance in face verification, in: proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1701–1708.
Wang KJ, GuoFeng Z, GuiXia F et al (2013) An approach to fast eye location and face plane rotation correction. Journal of Computer-Aided Design &Computer Graphics 25(6):865–872
Xi M, Liang C, Polajnar D, Tong W (2016) Local binary pattern network: A deep learning approach for face recognition. 2016 IEEE International Conference on Image Processing (ICIP):3224–3228
Xiao C, Ze L, Wang YL et al (2016) Face Recogniton with pose variation based on fusion of multi-scale MRF and SCT. Application Research of Computers 33(8):2519–2523
Xiao-hong Z Z-h H (2013) Linear locality preserving and discriminating projection for face recognition. J Elec Info Techn 35(2):463–467
Xiu-Juan C, Shi-Guang S, Xi-Lin C et al (2007) Local linear regression for pose invariant face recognition. IEEE Trans on Image Process 16(7):1716–1725
Yin X, Liu X (2018) Multi-task convolutional neural network for pose-invariant face recognition. IEEE Trans Image Process 27(2):964–975
Xi Yin, Xiang Yu, Kihyuk Sohn, et al. Feature Transfer Learning for Deep Face Recognition with Long-Tail Data. In Proceeding of IEEE Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, Jun. 2019. arXiv:1803.09014.
Yong C, Ting ting H, Hua lin L et al (2016) Multi-pose face ensemble classification aided by Gabor features and deep belief nets. Optik 127(2):946–954
Zhang ZC, Hong WC (2019) Electric load forecasting by complete ensemble empirical model decomposition adaptive noise and support vector regression with quantum-based dragonfly algorithm. Nonlinear Dynamics 98:1107–1136
Zhang Y-N, Guo Z, Xia Y et al (2012) 2D representation of facial surfaces for multi-pose 3D face recognition. Pattern Recogn Lett 33:530–536
Zhang H, Nasrabadi NM, Zhang Y et al (2012) Joint dynamic sparse representation for multi-view face recognition. Pattern Recogn 45:1290–1298
Zhang Y, Zhao D, Sun J, Zou G, Li W (2016) Adaptive convolutional neural network and its application in face recognition. Neural Process Lett 43(2):389–399
Zhao J, Yu C, Xu Y et al (2018) Towards Pose Invariant Face Recognition in the Wild. IEEE Conf Comp Vision Pattern Recogn (CVPR):2207–2216
Zou G-f, Gui-xia F, Gao M-l et al (2019) A novel construction method of convolutional neural network model based on data-driven. Multimed Tools Appl 78(6):6969–6987
Funding
This research was funded by the Visiting Project Funds of Shandong University of Technology, the Integration Funds of Shandong University of Technology and Zhangdian District (No.118228), the National Natural Science Foundation of China (No. 61601266, No.61801272), the Natural Science Foundation of Shandong Province of China (No. ZR2015FL029, ZR2016FL14) .
Author information
Authors and Affiliations
Contributions
Guofeng Zou proposed the original idea and wrote this paper; Guixia Fu analyzed the data and correct the translation; Mingliang Gao and Jinfeng Pan reviewed this paper; Zheng Liu participated in the rationality of the demonstration experiment.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zou, G., Fu, G., Gao, M. et al. A new approach for small sample face recognition with pose variation by fusing Gabor encoding features and deep features. Multimed Tools Appl 79, 23571–23598 (2020). https://doi.org/10.1007/s11042-020-09076-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09076-1