Elsevier

Pattern Recognition

Volume 75, March 2018, Pages 15-24
Pattern Recognition

Video-based kinship verification using distance metric learning

https://doi.org/10.1016/j.patcog.2017.03.001Get rights and content

Highlights

  • We investigate the problem of kinship verification from facial videos.

  • We present a new video face dataset for the video-based kinship verification study.

  • We develop a benchmark to evaluate state-of-the-art metric learning methods in video-based kinship verification.

  • Experiments show the efficacy of distance metric learning in kinship verification.

Abstract

In this paper, we investigate the problem of video-based kinship verification via human face analysis. While several attempts have been made on facial kinship verification from still images, to our knowledge, the problem of video-based kinship verification has not been formally addressed in the literature. In this paper, we make the two contributions to video-based kinship verification. On one hand, we present a new video face dataset called Kinship Face Videos in the Wild (KFVW) which were captured in wild conditions for the video-based kinship verification study, as well as the standard benchmark. On the other hand, we employ our benchmark to evaluate and compare the performance of several state-of-the-art metric learning based kinship verification methods. Experimental results are presented to demonstrate the efficacy of our proposed dataset and the effectiveness of existing metric learning methods for video-based kinship verification. Lastly, we also evaluate human ability on kinship verification from facial videos and experimental results show that metric learning based computational methods are not as good as that of human observers.

Introduction

Kinship verification from human faces in a relatively new problem in biometrics in recent years. The key motivation of this research topic is from the research observations in results in psychology and cognitive sciences [1], [2], [3], [4] where human faces convey an important cue for kin similarity measure because children usually look like their parents. Verifying human kinship relationship has several potential applications such as image annotation, family album organization, social media mining, and missing children searching. Over the past few years, a number of kinship verification methods have been proposed in the literature, which aims to present effective computational models to verify human kinship relations via facial image analysis [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17]. While these methods have achieved some encouraging performance [5], [6], [7], [8], [9], [10], [11], [12], [13], [15], [18], it is still challenging to develop discriminative and robust kinship verification approached for real-world applications, especially when face images are captured in unconstrained environments where large variations of pose, illumination, expression, and background occurs.

Most existing kinship verification methods determine human kinship relationship from still face images. Due to the large variations of human faces, a single still image may not be discriminative enough to verify human kin relationship. Compared to a single image, a face video provides more information to describe the appearance of human face. It can capture the face of the person of interest from different poses, expressions, and illuminations. Moreover, face videos can be much easier captured in real applications because there are extensive surveillance cameras installed in public areas. Hence, it is desirable to employ face videos to determine the kin relations of persons. However, it is also challenging to exploit discriminative information of face videos because intra-class variations are usually larger within a face video than a single sill image.

In this paper, we investigate the problem of video-based kinship verification via human face analysis. Specifically, we make the two contributions to video-based kinship verification. On one hand, we present a new video face dataset called Kinship Face Videos in the Wild (KFVW) which were captured in wild conditions for the video-based kinship verification study, as well as the standard benchmark. On the other hand, we employ our benchmark to evaluate and compare the performance of several state-of-the-art metric learning based kinship verification methods. Experimental results are presented to demonstrate the efficacy of our proposed dataset and the effectiveness of existing metric learning methods for video-based kinship verification. Lastly, we also test human ability on kinship verification from facial videos and experimental results show that metric learning based computational methods are not as good as that of human observers.

The rest of this paper is organized as follows. In Section 2, we briefly review some related work, and Section 3 introduces the Kinship Face Videos in the Wild (KFVW) dataset. Section 4 presents some popular metric learning methods which have been widely used in kinship verification. Section 5 presents the experimental results and analysis. Finally, Section 6 concludes this paper.

Section snippets

Related work

In this section, we briefly review the related topics to our work: (1) kinship verification, (2) metric learning, and (3) video-based face analysis.

The kinship face videos in the wild dataset

In past few years, several facial datasets have been released to advance the kinship verification problem, e.g., CornellKin [5], UB KinFace [7], IIITD Kinship [18], Family101 [12], KinFaceW-I [13], KinFaceW-II [13], TSKinFace [66], etc. Table 2 provides a summary of existing facial datasets for kinship verification. However, these datasets only consist of still face images, in which each subject usually has a single face image. Due to the large variations of human faces, a single still image

Video-based kinship verification using metric learning

Metric learning involves seeking a suitable distance metric from a training set of data points. Following the evaluation and settings used in Ref. [67], we employ several distance metric learning methods as baseline methods for the video-based kinship verification problem. These metric learning methods include information theoretic metric learning (ITML) [68], side-information based linear discriminant analysis (SILD) [69], KISS metric learning (KISSME) [28], and cosine similarity metric

Experiments

In this section, we evaluated several state-of-the-art metric learning methods for video-based kinship verification on the KFVW dataset, and provided some baseline results on this dataset.

Conclusion

In this paper, we have studied the problem of video-based kinship verification. To our best knowledge, this problem has not been formally addressed in the literature. We have first presented a new video face dataset called Kinship Face Videos in the Wild (KFVW) which were captured in wild conditions for the video-based kinship verification study, as well as the standard benchmark. Then, we have evaluated and compared the performance of several state-of-the-art metric learning based kinship

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61603048, the Beijing Natural Science Foundation under Grant 4174101, and the Fundamental Research Funds for the Central Universities.

Haibin Yan received the B.E. and M.E. degrees from the Xi’an University of Technology, Xi’an, China, in 2004 and 2007, and the Ph.D. degree from the National University of Singapore, Singapore, in 2013, all in mechanical engineering. Now, she is an Assistant Professor in the School of Automation, Beijing University of Posts and Telecommunications, Beijing, China. From October 2013 to July 2015, she was a research fellow at the Department of Mechanical Engineering, National University of

References (73)

  • G. Guo et al.

    Kinship measurement on salient facial features

    IEEE Trans. Instrum. Meas.

    (2012)
  • X. Zhou et al.

    Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments

    ACM International Conference on Multimedia

    (2012)
  • G. Somanath et al.

    Can faces verify blood-relations?

    IEEE International Conference on Biometrics: Theory, Applications and Systems

    (2012)
  • R. Fang et al.

    Kinship classification by modeling facial feature heredity

    IEEE International Conference on Image Processing

    (2013)
  • J. Lu et al.

    Neighborhood repulsed metric learning for kinship verification

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2014)
  • Y. Guo et al.

    Graph-based kinship recognition

    International Conference on Pattern Recognition

    (2014)
  • H. Yan et al.

    Discriminative multimetric learning for kinship verification

    IEEE Trans. Inf. Forensics Secur.

    (2014)
  • J. Lu et al.

    Kinship verification in the wild: the first kinship verification competition

    IEEE International Joint Conference on Biometrics

    (2014)
  • N. Kohli et al.

    Self-similarity representation of weber faces for kinship classification

    IEEE International Conference on Biometrics: Theory, Applications, and Systems

    (2012)
  • H. Dibeklioglu et al.

    Like father, like son: facial expression dynamics for kinship verification

    IEEE International Conference on Computer Vision

    (2013)
  • H. Yan et al.

    Prototype-based discriminative feature learning for kinship verification

    IEEE Trans. Cybern.

    (2015)
  • S. Xia et al.

    Kinship verification through transfer learning

    International Joint Conference on Artificial Intelligence

    (2011)
  • M. Shao et al.

    Genealogical face recognition based on ub kinface database

    IEEE Conference on Computer Vision and Pattern Recognition Workshops

    (2011)
  • J. Hu et al.

    Large margin multi-metric learning for face and kinship verification in the wild

    Asian Conference on Computer Vision

    (2014)
  • S. Du et al.

    Improved face representation by nonuniform multilevel selection of Gabor convolution features

    IEEE Trans. Syst. Man Cybern. Part B

    (2009)
  • W. Deng et al.

    Extended src: undersampled face recognition via intraclass variant dictionary

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2012)
  • M. Guillaumin et al.

    Is that you? Metric learning approaches for face identification

    IEEE International Conference on Computer Vision

    (2009)
  • J. Lu et al.

    Regularized locality preserving projections and its extensions for face recognition

    IEEE Trans. Syst. Man Cybern. Part B

    (2010)
  • M. Köstinger et al.

    Large scale metric learning from equivalence constraints

    IEEE Conference on Computer Vision and Pattern Recognition

    (2012)
  • D. Tran et al.

    Human activity recognition with metric learning

    European Conference on Computer Vision

    (2008)
  • B. Xiao et al.

    Learning distance metric for regression by semidefinite programming with application to human age estimation

    ACM International Conference on Multimedia

    (2009)
  • W.-S. Zheng et al.

    Person re-identification by probabilistic relative distance comparison

    IEEE Conference on Computer Vision and Pattern Recognition

    (2011)
  • A. Mignon et al.

    Pcca: a new approach for distance learning from sparse pairwise constraints

    IEEE Conference on Computer Vision and Pattern Recognition

    (2012)
  • J. Hu et al.

    Discriminative deep metric learning for face verification in the wild

    IEEE Conference on Computer Vision and Pattern Recognition

    (2014)
  • J. Lu et al.

    Human identity and gender recognition from gait sequences with arbitrary walking directions

    IEEE Trans. Inf. Forensics Secur.

    (2014)
  • J. Lu et al.

    Discriminative multimanifold analysis for face recognition from a single training sample per person

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2013)
  • Cited by (0)

    Haibin Yan received the B.E. and M.E. degrees from the Xi’an University of Technology, Xi’an, China, in 2004 and 2007, and the Ph.D. degree from the National University of Singapore, Singapore, in 2013, all in mechanical engineering. Now, she is an Assistant Professor in the School of Automation, Beijing University of Posts and Telecommunications, Beijing, China. From October 2013 to July 2015, she was a research fellow at the Department of Mechanical Engineering, National University of Singapore, Singapore. Her research interests include service robotics and computer vision.

    Junlin Hu received the B.E. degree from the Xian University of Technology, Xian, China, in 2008, and the M.E. degree from the Beijing Normal University, Beijing, China, in 2012. He is currently pursuing the Ph.D. degree with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. His current research interests include computer vision, pattern recognition, and biometrics.

    View full text