Abstract
The face is one of the most reliable and easy-to-acquire biometric features, widely used for the recognition of individuals. In controlled environments facial recognition systems are highly effective, however, in real world scenarios and under varying lighting conditions, pose changes, facial expressions, occlusions and low resolution of captured images/videos, the task of recognizing faces becomes significantly complex. In this context it has been shown that certain attributes can be retrieved with a relative probability of success, being useful to complement a non conclusive result of a biometric system. In this paper we present an overview on face describable visual attributes and in particular of the so-called soft biometrics (e.g., facial marks, gender, age, skin color, and other physical characteristics). We review core issues regarding this topic, for instance what are the soft biometrics, which of them are the most robust in video surveillance and other uncontrolled scenarios, how different approaches have been addressed in the literature for their representation and classification, which datasets can be used for evaluation, which related problems remain unresolved and which are the possible ways to approach them.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Antipov G, Baccouche M, Berrani SA, Dugelay JL (2017) Effective training of convolutional neural networks for face-based gender and age prediction. Pattern Recognit (PR) 72(C):15–26
Awwad A, Ahmad A, Walid AS (2013) Arabic race classification of face images. Int J Comput Technol 4(2):234–239
Becerra-Riera F, Méndez-Vázquez H, Morales-González A, Tistarelli M (2017a) Age and gender classification using local appearance descriptors from facial components. In: International joint conference on biometrics (IJCB). IEEE, Piscataway, pp 799–804
Becerra-Riera F, Morales-Gonález A, Méndez-Vázquez H (2017b) Facial marks for improving face recognition. Pattern Recognit Lett (PRLetters) 113:3–9
Becerra-Riera F, Morales-González A, Méndez-Vázquez H (2018) Exploring local deep representations for facial gender classification in videos. In: International workshop on artificial intelligence and pattern recognition (IWAIPR). Lecture notes in computer science, vol 11047. Springer, Berlin, pp 104–112
Bekhouche SE, Ouafi A, Dornaika F, Taleb-Ahmed A, Hadid A (2017) Pyramid multi-level features for facial demographic estimation. Expert Syst Appl 80(C):297–310
Bekios-Calfa J, Buenaposada JM, Baumela L (2014) Robust gender recognition by exploiting facial attributes dependencies. Pattern Recognit Lett (PRLetters) 36:228–234
Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. Trans Pattern Anal Mach Intell (TPAMI) 35(8):1798–1828
Beveridge JR, Phillips PJ, Bolme DS, Draper BA, Givens GH, Lui YM, Teli MN, Zhang H, Scruggs WT, Bowyer KW, Flynn PJ, Cheng S (2013) The challenge of face recognition from digital point-and-shoot cameras. In: BTAS. IEEE, Piscataway, pp 1–8
Bruce V, Burtonu AM, Hanna E, Healey P, Mason O, Coombes A, Fright R, Linney A (1993) Sex discrimination: how do we tell the difference between male and female faces? Perception 22:131–152
Brunelli R, Poggio T (1993) Face recognition: features versus templates. Trans Pattern Anal Mach Intell (TPAMI) 15(10):1042–1052
Canziani A, Paszke A, Culurciello E (2017) An analysis of deep neural network models for practical applications. In: International symposium on circuits and systems
Carcagnì P, Coco MD, Cazzato D, Leo M, Distante C (2015) A study on different experimental configurations for age, race, and gender estimation problems. EURASIP J Image Video Process 2015(1):37
Carnicky J, Chorvat J (2006) Three-dimensional measurement of human face with structured-light illumination. Meas Sci Rev 6:1–4
Castrillón-Santana M, Lorenzo-Navarro J, Ramón-Balmaseda E (2013) Improving gender classification accuracy in the wild. In: 18th Iberoamerican congress on pattern recognition (CIARP). Springer, Berlin, pp 270–277
Castrillón-Santana M, Lorenzo-Navarro J, Ramón-Balmaseda E (2017) Descriptors and regions of interest fusion for in- and cross-database gender classification in the wild. Image Vis Comput 57(C):15–24
Castrillón-Santana M, Marsico MD, Nappi M, Riccio D (2017) Meg: texture operators for multi-expert gender classification. Comput Vis Image Underst 156:4–18
Chen BC, Chen CS, Hsu WH (2014) Cross-age reference coding for age-invariant face recognition and retrieval. In: European conference on computer vision (ECCV). Lecture notes in computer science, vol 8694. Springer, Berlin, pp 768–783
Chen BC, Chen YY, Kuo YH, Hsu WH (2013) Scalable face image retrieval using attribute-enhanced sparse codewords. Trans Multimed 15:1163–1173
Dago-Casas P, D.González-Jiménez Long-Yu L, Alba-Castro JL (2011) Single- and cross-database benchmarks for gender classification under unconstrained settings. In: International conference on computer vision (ICCV) workshops. IEEE, Piscataway, pp 2152–2159
Dantcheva A, Elia P, Ross A (2016) What else does your biometric data reveal? A survey on soft biometrics. Trans Inf Forensics Secur (TIFS) 11(3):441–467
Dantcheva A, Velardo C, D’angelo A, Dugelay JL (2011) Bag of soft biometrics for person identification: new trends and challenges. Mutimed Tools Appl 51(2):739–777
Demirkus M, Precup D, Clark JJ, Arbel T (2015) Hierarchical temporal graphical model for head pose estimation and subsequent attribute classification in real-world videos. Comput Vis Image Underst 136:128–145
Demirkus M, Precup D, Clark JJ, Arbel T (2016) Hierarchical spatio-temporal probabilistic graphical model with multiple feature fusion for binary facial attribute classification in real-world face videos. IEEE Trans Pattern Anal Mach Intell 38(6):1185–1203
Deng Y, Luo P, Loy CC, Tang X (2014) Pedestrian attribute recognition at far distance. In: 22nd International conference on multimedia, MM ’14. ACM, New York, pp 789–792
Eidinger E, Enbar R, Hassner T (2014) Age and gender estimation of unfiltered faces. Trans Inf Forensics Secur (TIFS) 9(12):2170–2179
Eldib MY, El-Saban M (2010) Human age estimation using enhanced bio-inspired features (ebif). In: International conference on image processing (ICIP), pp 1589–1592
Eldib MY, Onsi HM (2011) Human age estimation framework using different facial parts. Egypt Inform J 12(1):53–59
Farkas LG (1994) Anthropometry of the head and face. Raven Press, Dhaka
Fellous JM (1997) Gender discrimination and prediction on the basis of facial metric information. Vis Res 37(14):1961–1973
Fu S, He H, Hou ZG (2014) Learning race from face: a survey. Trans Pattern Anal Mach Intell (TPAMI) 36(12):2483–2509
Fu Y, Guo G, Huang TS (2010) Age synthesis and estimation via faces: a survey. Trans Pattern Anal Mach Intell (TPAMI) 32(11):1955–1976
Gallagher A, Chen T (2009) Understanding images of groups of people. In: Conference on computer vision and pattern recognition(CVPR), vol 00, pp 256–263
Geng X, Zhou ZH, Smith-Miles K (2007) Automatic age estimation based on facial aging patterns. Trans Pattern Anal Mach Intell (TPAMI) 29(12):2234–2240
González-Sosa E, Fiérrez J, Vera-Rodríguez R, Alonso-Fernández F (2018) Facial soft biometrics for recognition in the wild: recent works, annotation and cots evaluation. Trans Inf Forensics Secur (TIFS) 13(7):2001–2014
Gunay A, Nabiyev VV (2008) Automatic age classification with lbp. In: International symposium on computer and information sciences (ISCIS). IEEE, Piscataway
Guo G, Dyer CR, Fu Y, Huang TS (2009) Is gender recognition affected by age?. IEEE, Piscataway, pp 2032–2039
Guo G, Mu G (2010) A study of large-scale ethnicity estimation with gender and age variations. In: Conference on computer vision and pattern recognition (CVPR) workshops. IEEE, Piscataway, pp 79–86
Han H, Jain AK (2014) Age, gender and race estimation from unconstrained face images
Han H, Jain AK, Shan S, Chen X (2017) Heterogeneous face attribute estimation: a deep multi-task learning approach. Trans Pattern Anal Mach Intell (TPAMI) 40:2597–2609
Han H, Otto C, Liu X, Jain AK (2015) Demographic estimation from face images: human vs. machine performance. Trans Pattern Anal Mach Intell (TPAMI) 37(6):1148–1161
Hu SYD, Jou B, Jaech A, Savvides M (2011) Fusion of region-based representations for gender identification. In: International joint conference on biometrics (IJCB). IEEE, Piscataway, pp 1–7
Huang D, Ding H, Wang C, Wang Y, Zhang G, Chen L (2014) Local circular patterns for multi-modal facial gender and ethnicity classification. Image Vis Comput 32(12):1181–1193
Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Tech Rep. University of Massachusetts, Amherst, pp 07-49
Huang T, Xiong Z, Zhang Z (2005) Face recognition applications. Springer, New York, pp 371–390
Huang Z, Shan S, Wang R, Zhang H, Lao S, Kuerban A, Chen X (2015) A benchmark and comparative study of video-based face recognition on cox face database. IEEE Trans Image Process 24(12):5967–5981
Jain A, Huang J (2004) Integrating independent components and linear discriminant analysis for gender classification. In: 6th International conference on automatic face and gesture recognition (FGR). IEEE, Piscataway, pp 159–163
Jain AK, Dass SC, Nandakumar K, N K (2004a) Can soft biometric traits assist user recognition? In: Biometric technology for human identification, vol 5404. SPIE, Bellingham, pp 561–572
Jain AK, Dass SC, Nandakumar K, N K (2004b) Soft biometric traits for personal recognition systems. In: International conference on biometric authentication, pp 731–738
Jain AK, Klare B, Park U (2011) Face recognition: some challenges in forensics. In: 9th International conference on automatic face and gesture recognition and workshops (FGR). IEEE, Piscataway, pp 726–733
Jain AK, Ross A, Prabhakar S (2004) An introduction to biometric recognition. Trans Circuits Syst Video Technol 14(1):4–20
Jia S, Cristianini N (2015) Learning to classify gender from four million images. Pattern Recognit Lett (PRLetters) 58(C):35–41
Jr KR, Tesafaye T (2006) Morph: a longitudinal image database of normal adult age-progression. In: 7th International conference on automatic face and gesture recognition (FGR). IEEE, Piscataway, pp 341–345
Juefei-Xu F, Verma E, Goel P, Cherodian A, Savvides M (2016) Deepgender: occlusion and low resolution robust facial gender classification via progressively trained convolutional neural networks with attention. In: Conference on computer vision and pattern recognition (CVPR) workshops. IEEE, Piscataway, pp 136–145
Khan A, Majid A, Mirza AM (2005) Combination and optimization of classifiers in gender classification using genetic programming. Int J Knowl-Based Intell Eng Syst (IJKBIES) 9(1):1–11
Khan R, Hanbury A, Stöttinger J, Bais A (2012) Color based skin classification. Pattern Recognit Lett (PRLetters) 33(2):157–163
Kim M, Cheeyong K (2015) Augmented reality fashion apparel simulation using a magic mirror. Int J Smart Home 9(2):169–178
Kumar N, Belhumeur PN, Nayar SK (2008) Facetracer: A search engine for large collections of images with faces. In: 10th European conference on computer vision (ECCV). Lecture notes in computer science, vol 5305. Springer, Berlin, pp 340–353
Kumar N, Berg AC, Belhumeur PN, Nayar SK (2009) Attribute and simile classifiers for face verification. In: 12th International conference on computer vision (ICCV). IEEE, Piscataway, pp 365–372
Kumar N, Berg AC, Belhumeur PN, Nayar SK (2011) Describable visual attributes for face verification and image search. Trans Pattern Anal Mach Intell (TPAMI) 33(10):1962–1977
Kwon YH, da Vitoria Lobo N (1999) Age classification from facial images. Comput Vis Image Underst 74(1):1–21
Lanitis A, Taylor CJ, Cootes TF (2002) Toward automatic simulation of aging effects on face images. Trans Pattern Anal Mach Intell (TPAMI) 24(4):442–455
Levi G, Hassner T (2015) Age and gender classification using convolutional neural networks. In: Conference on computer vision and pattern recognition (CVPR) workshops. IEEE, Piscataway, pp 34–42
Li SZ, Jain A (2015) Encyclopedia of biometrics. Springer, Berlin
Li T, Chang H, Wang M, Ni B, Hong R, Yan S (2015) Crowded scene analysis: a survey. Trans Circuits Syst Video Technol 25:367–386
Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. Trans Image Process 11(4):467–476
Liu H, Lu J, Feng J, Zhou J (2017) Ordinal deep feature learning for facial age estimation. In: 12th International conference on automatic face and gesture recognition (FGR). IEEE, Piscataway, pp 157–164
Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: International conference on computer vision (ICCV). IEEE, Piscataway, pp 3730–3738
Lu X, Jain AK (2004) Ethnicity identification from face images. In: Biometric technology for human identification, vol 5404. SPIE, Bellingham, pp 114–123
Lyons MJ, Budynek J, Plante A, Akamatsu S (2000) Classifying facial attributes using a 2-d gabor wavelet representation and discriminant analysis. In: 4th International conference on automatic face and gesture recognition (FGR), pp 202–207
Manesh FS, Ghahramani M, Tan YP (2010) Facial part displacement effect on template-based gender and ethnicity classification. In: 11th International conference on control automation robotics vision (ICARCV). IEEE, Piscataway, pp 1644–1649
Mansanet J, Albiol A, Paredes R (2016) Local deep neural networks for gender recognition. Pattern Recognit Lett (PRLetters) 70(C):80–86
Martínez A, Benavente R (1998) The ar face database. In: Tech Rep 24. Computer Vision Center, Barcelona
Merkow J, Jou B, Savvides M (2010) An exploration of gender identification using only the periocular region. In: 4th International conference on biometrics: theory applications and systems (BTAS). IEEE, Piscataway, pp 1–5
Microsoft: how old do i look? (2015). https://how-old.net/
Minear M, Park DC (2004) A lifespan database of adult facial stimuli. Behav Res Methods, Instrum Comput 36(4):630–633
Moeini A, Moeini H, Safai AM, Faez K (2017) Regression facial attribute classification via simultaneous dictionary learning. Pattern Recognit (PR) 62:99–113
Mozaffari S, Behravan H, Akbari R (2010) Gender classification using single frontal image per person: combination of appearance and geometric based features. In: 20th International conference on pattern recognition (ICPR). IEEE, Piscataway, pp 1192–1195
Muhammad G, Hussain M, Alenezy F, Bebis G, Mirza AM, Aboalsamh H (2012) Race classification from face images using local descriptors. Int J Artif Intell Tools 21(5):1250019
Muhammad G, Hussain M, Alenezy F, Bebis G, Mirza AM, Aboalsamh H (2012) Race recognition from face images using weber local descriptor. In: 19th International conference on systems, signals and image processing (IWSSIP), pp 421–424
Nixon M (1985) Eye spacing measurement for facial recognition. SPIE, Bellingham, pp 279–283
Ojala T, Pietikainen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recognit (PR) 29(1):51–59
Ou Y, Wu X, Qian H, Xu Y (2005) A real time race classification system. In: International conference on information acquisition. IEEE, Piscataway
Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Proceedings of the British Machine Vision Conference 2015. BMVC, Swansea, pp 41.1–41.12, 7–10 Sept 2015
Phillips PJ, Wechsler H, Huang J, Rauss PJ (1998) The feret database and evaluation procedure for face-recognition algorithms. Image Vis Comput 16(5):295–306
Radu I (2014) Augmented reality in education: a meta-review and cross-media analysis. Pers Ubiquitous Comput 18(6):1533–1543
Ramanathan N, Chellappa R (2006) Modeling age progression in young faces. Conf Comput Vis Pattern Recognit (CVPR) 1:387–394
Rhodes H (1956) Alphonse Bertillon: father of scientific detection. Abelard-Schuman, Bishopbriggs
Riccio D, Tortora G, Marsico MD, Wechsler H (2012) Ega-ethnicity, gender and age, a pre-annotated face database. In: Workshop on biometric measurements and systems for security and medical applications (BIOMS). IEEE, Piscataway, pp 1–8
Rodríguez P, Cucurull G, Gonfaus JM, Roca FX, González J (2017) Age and gender recognition in the wild with deep attention. Pattern Recognit (PR) 72(C):563–571
Roomi SMM, Virasundarii SL, Selvamegala S, Jeevanandhame S, Hariharasudhan D (2011) Race classification based on facial features. In: 3rd National conference on computer vision, pattern recognition, image processing and graphics (NCVPRIPG). IEEE, Piscataway, pp 54–57
Rothe R, Timofte R, Gool LV (2016) Deep expectation of real and apparent age from a single image without facial landmarks. Int J Comput Vis (IJCV) 126(2–4):144–157
Sahoo TK, Banka H (2017) New hybrid pca-based facial age estimation using inter-age group variation-based hierarchical classifier. Arab J Sci Eng 42(8):3337–3355
Salah SH, Du H, Al-Jawad N (2013) Fusing local binary patterns with wavelet features for ethnicity identification. Int J Comput, Inf, Syst Control Eng 7:330–336
Scholz J, Smith AN (2016) Augmented reality: designing immersive experiences that maximize consumer engagement. Bus Horiz 59(2):149–161
Selim M, Sundararajan S, Pagani A, Stricker D (2018) Image quality-aware deep networks ensemble for efficient gender recognition in the wild. In: Proceedings of the 13th international joint conference on computer vision, imaging and computer graphics theory and applications (VISIGRAPP), vol 5
Semertzidis T, Axenopoulos A, Karadimos P, Daras P (2016) Soft biometrics in low resolution and low quality cctv videos. In: 7th International conference on imaging for crime detection and prevention (ICDP). IET, Stevenage
Shan C (2012) Learning local binary patterns for gender classification on real world face images. Pattern Recognit Lett (PRLetters) 33(4):431–437
Amilia Sindi, Dwi Mahmud, Sulistiyo RND (2015) Face image-based gender recognition using complex-valued neural network. In: 3rd International conference on information and communication technology (ICoICT)
Stoyanova J, Georgieva P, Brito PQ, Milanova M (2015) Comparison of consumer purchase intention between interactive and augmented reality shopping platforms through statistical analyses. In: International symposium on innovations in intelligent sysTems and applications (INISTA), pp 62–69
Tapia JE, Perez CA (2013) Gender classification based on fusion of different spatial scale features selected by mutual information from histogram of lbp, intensity, and shape. Trans Inf Forensics Secur (TIFS) 8(3):488–499
Thida M, Yong YL, Climent-Pérez P, Lung Eng H, Remagnino P (2013) A literature review on video analytics of crowded scenes. Springer, Berlin, pp 17–36
Tin HHK, Sein MM (2011) Race identification for face images 1(02)
Tistarelli M, Yadav D, Vatsa M, Singh R (2013) Short and long-time ageing effects in face recognition. Age Factors Biom Process, Chapter
Tome P, Vera-Rodríguez R, Fiérrez J, Ortega-García J (2015) Facial soft biometric features for forensic face recognition. Forensic Sci Int 257:271–284
Wang JG, Li J, Yau WY, Sung E (2010) Boosting dense sift descriptors and shape contexts of face images for gender recognition. In: Conference on computer vision and pattern recognition (CVPR) workshops, pp 96–102
Wang W, He F, Zhao Q (2016) Facial ethnicity classification with deep convolutional neural networks. In: Chinese conference on biometric recognition (CCBR). Lecture notes in computer science, vol 9967, pp 176–185
Wang X, Guo R, Kambhamettu C (2015) Deeply-learned feature for age estimation. In: Winter conference on applications of computer vision (WACV), pp 534–541
Wild HA, Barrett SE, Spence MJ, O’Toole AJ, Cheng YD, Brooke J (2000) Recognition and sex categorization of adults’ and children’s faces: examining performance in the absence of sex-stereotyped cues. J Exp Child Psychol 77(4):269–291
Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: CVPR. IEEE Computer Society, Washington, DC, pp 529–534
Xia B, Sun H, Lu BL (2008) Multi-view gender classification based on local gabor binary mapping pattern and support vector machines. In: International joint conference on neural networks (IJCNN). IEEE, Piscataway, pp 3388–3395
Yang Z, Ai H (2007) Demographic classification with local binary patterns. In: Lecture notes in computer science. Springer, Berlin, pp 464–473
Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: 7th International conference on automatic face and gesture recognition (FGR). IEEE, Piscataway, pp 211–216
Ylioinas J, Hadid A, Pietikäinen M (2012) Age classification in unconstrained conditions using lbp variants. In: 21st International conference on pattern recognition (ICPR). IEEE, Piscataway, pp 1257–1260
Zhang L, Yang M, Feng X (2011) Sparse representation or collaborative representation: Which helps face recognition? In: International conference on computer vision (ICCV). IEEE, Piscataway, pp 471–478
Zhang N, Paluri M, Ranzato M, Darrell T, Bourdev LD (2014) Panda: Pose aligned networks for deep attribute modeling. In: Conference on computer vision and pattern recognition (CVPR). IEEE, Piscataway, pp 1637–1644
Zhong Y, Sullivan J, Li H (2016) Face attribute prediction using off-the-shelf deep learning networks. In: International conference on biometrics (ICB). IEEE, Piscataway
Zhu J, Liao S, Lei Z, Yi D, Li SZ (2013) Pedestrian attribute classification in surveillance: database and evaluation. In: International conference on computer vision workshops, ICCVW ’13. IEEE, Piscataway, pp 331–338
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Becerra-Riera, F., Morales-González, A. & Méndez-Vázquez, H. A survey on facial soft biometrics for video surveillance and forensic applications. Artif Intell Rev 52, 1155–1187 (2019). https://doi.org/10.1007/s10462-019-09689-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-019-09689-5