Skip to main content
SpringerLink
Log in

Kinship recognition from faces using deep learning with imbalanced data

  • Track 3: Biometrics and HCI
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Kinship verification from faces aims to determine whether two person share some family relationship based only on the visual facial patterns. This has attracted a significant interests among the scientific community due to its potential applications in social media mining and finding missing children. In this work, We propose a novel pattern analysis technique for kinship verification based on a new deep learning-based approach. More specifically, given a pair of face images, we first use Resnet50 to extract deep features from each image. Then, feature distances between each pair of images are computed. Importantly, to overcome the problem of unbalanced data, One Hot Encoding for labels is utilised. The distances finally are fed to a deep neural networks to determine the kinship relation. Extensive experiments are conducted on FIW dataset containing 11 classes of kinship relationships. The experiments showed very promising results and pointed out the importance of balancing the training dataset. Moreover, our approach showed interesting ability of generalization. Results show that our approach performs better than all existing approaches on grandparents-grandchildren type of kinship. To support the principle of open and reproducible research, we are soon making our code publicly available to the research community: github.com/Steven-HDQ/Kinship-Recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Aliradi R, Belkhir A, Ouamane A, Elmaghraby AS (2018) Dieda: discriminative information based on exponential discriminant analysis combined with local features representation for face and kinship verification. Multimed Tools Appl 1–18

  2. Almuashi M, Hashim SZM, Mohamad D, Alkawaz MH, Ali A (2017) Automated kinship verification and identification through human facial images: a survey. Multimed Tools Appl 76(1):265–307

    Article  Google Scholar 

  3. Bisogni C, Narducci F (2022) Kinship recognition: how far are we from viable solutions in smart environments? Procedia Computer Science 198:225–230

    Article  Google Scholar 

  4. Chen X, An L, Yang S, Wu W (2017) Kinship verification in multi-linear coherent spaces. Multim Tools Appl 76(3):4105–4122. https://doi.org/10.1007/s11042-015-2930-9

    Article  Google Scholar 

  5. Dehghan A, Ortiz EG, Villegas R, Shah M (2014) Who do I look like? determining parent-offspring resemblance via gated autoencoders. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp 1757–1764. IEEE Computer Society, DOI https://doi.org/10.1109/CVPR.2014.227, (to appear in print)

  6. Fang R, Gallagher AC, Chen T, Loui AC (2013) Kinship classification by modeling facial feature heredity. In: IEEE international conference on image processing, ICIP 2013, Melbourne, Australia, September 15-18, 2013, pp 2983–2987. IEEE, DOI https://doi.org/10.1109/ICIP.2013.6738614, (to appear in print)

  7. Fang R, Tang KD, Snavely N, Chen T (2010) Towards computational models of kinship verification. In: Proceedings of the international conference on image processing, ICIP 2010, September 26-29, Hong Kong, China, pp 1577–1580. IEEE, DOI https://doi.org/10.1109/ICIP.2010.5652590, (to appear in print)

  8. Goyal A, Meenpal T (2020) Patch-based dual-tree complex wavelet transform for kinship recognition. IEEE Trans Image Process 30:191–206

    Article  Google Scholar 

  9. Goyal A, Meenpal T (2021) Eccentricity based kinship verification from facial images in the wild. Pattern Anal Applic 24(1):119–144

    Article  Google Scholar 

  10. Guo Y, Dibeklioglu H, van der Maaten L (2014) Graph-based kinship recognition. In: 22Nd international conference on pattern recognition, ICPR 2014, Stockholm, Sweden, August 24-28, 2014, pp 4287–4292. IEEE Computer Society, DOI https://doi.org/10.1109/ICPR.2014.735, (to appear in print)

  11. Hazourli AR, Djeghri A, Salam H, Othmani A (2021) Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display. Multimed Tool Appl 80(9):13,639–13,662

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 770–778. IEEE Computer Society, DOI https://doi.org/10.1109/CVPR.2016.90, (to appear in print)

  13. Hu J, Lu J, Yuan J, Tan Y (2014) Large margin multi-metric learning for face and kinship verification in the wild. In: Cremers D, Reid ID, Saito H, Yang M (eds) Computer Vision - ACCV 2014 - 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part III, Lecture Notes in Computer Science, vol 9005, pp 252–267. Springer, DOI https://doi.org/10.1007/978-3-319-16811-1_17, (to appear in print)

  14. Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: a database forStudying face recognition in unconstrained environments. In: Workshop on faces in ’real-life’ images: detection, alignment, and recognition. Erik Learned-Miller and Andras Ferencz and Fredéric Jurié, Marseille, France. https://hal.inria.fr/inria-00321923

  15. Laiadi O, Ouamane A, Benakcha A, Taleb-Ahmed A, Hadid A (2021) A weighted exponential discriminant analysis through side-information for face and kinship verification using statistical binarized image features. Int J Mach Learn Cybern 12(1):171–185

    Article  Google Scholar 

  16. Laiadi O, Ouamane A, Boutellaa E, Benakcha A, Taleb-ahmed A, Hadid A (2019) Kinship verification from face images in discriminative subspaces of color components. Multim Tool Appl 78(12):16,465–16,487. https://doi.org/10.1007/s11042-018-7027-9

    Article  Google Scholar 

  17. Li Y, Zeng J, Zhang J, Dai A, Kan M, Shan S, Chen X (2017) Kinnet: fine-to-coarse deep metric learning for kinship verification. In: Proceedings of the 2017 Workshop on Recognizing Families In the Wild, RFIW ’17, pp 13–20, Association for Computing Machinery, New York, NY, USA, DOI https://doi.org/10.1145/3134421.3134425, (to appear in print)

  18. Lu J, Hu J, Tan Y (2017) Discriminative deep metric learning for face and kinship verification. IEEE Trans Image Process 26(9):4269–4282. https://doi.org/10.1109/TIP.2017.2717505

    Article  MathSciNet  MATH  Google Scholar 

  19. Lu J, Hu J, Zhou X, Zhou J, Santana MC, Lorenzo-navarro J, Kou L, Shang Y, Bottino A, Vieira TF (2014) Kinship verification in the wild: The first kinship verification competition. In: IEEE international joint conference on biometrics, clearwater, IJCB 2014, FL, USA, September 29 - October 2, 2014, pp 1–6. IEEE, DOI https://doi.org/10.1109/BTAS.2014.6996230, (to appear in print)

  20. Mukherjee M, Meenpal T, Goyal A (2022) Fusekin: weighted image fusion based kinship verification under unconstrained age group. J Vis Commun Image Represent 84(103):470

    Google Scholar 

  21. Nader N, El-Gamal FEZ, El-Sappagh S, Kwak KS, Elmogy M (2021) Kinship verification and recognition based on handcrafted and deep learning feature-based techniques. PeerJ Computer Science e735:7

    Google Scholar 

  22. Othmani A, Taleb AR, Abdelkawy H, Hadid A (2020) Age estimation from faces using deep learning: a comparative analysis. Comput Vis Image Underst 196(102):961

    Google Scholar 

  23. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  24. Qin X, Tan X, Chen S (2015) Tri-subjects kinship verification: Understanding the core of a family. In: 14th IAPR international conference on machine vision applications, MVA 2015, Miraikan, Tokyo, Japan, 18-22 May, 2015, pp 580–583. IEEE, DOI https://doi.org/10.1109/MVA.2015.7153258, (to appear in print)

  25. Robinson JP, Shao M, Fu Y (2020) Visual kinship recognition: a decade in the making. CoRR 2006.16033

  26. Robinson JP, Shao M, Wu Y, Fu Y (2016) Families in the wild (FIW): large-scale kinship image database and benchmarks. In: Hanjalic A, Snoek C, Worring M, Bulterman DCA, Huet B, Kelliher A, Kompatsiaris Y, Li J (eds) Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016, pp 242–246. ACM, DOI https://doi.org/10.1145/2964284.2967219, (to appear in print)

  27. Robinson JP, Shao M, Wu Y, Liu H, Gillis T, Fu Y (2018) Visual kinship recognition of families in the wild. IEEE Trans Pattern Anal Mach Intell 40(11):2624–2637. https://doi.org/10.1109/TPAMI.2018.2826549

  28. Robinson JP, Yin Y, Khan Z, Shao M, Xia S, Stopa M, Timoner S, Turk MA, Chellappa R, Fu Y (2020) Recognizing families in the wild (RFIW): the 4th edition. In: 15th IEEE international conference on automatic face and gesture recognition, FG 2020, Buenos Aires, Argentina, November 16-20, 2020, pp 857–862. IEEE, DOI https://doi.org/10.1109/FG47880.2020.00138, (to appear in print)

  29. Somasundaram A, U SR (2016) Data imbalance: effects and solutions for classification of large and highly imbalanced data.. In: international conference on research in engineering, computers and technology (ICRECT 2016).

  30. Schoneveld L, Othmani A (2021) Towards a general deep feature extractor for facial expression recognition 2021 IEEE international conference on image processing (ICIP), pp 2339–2342, DOI https://doi.org/10.1109/ICIP42928.2021.9506025, (to appear in print)

  31. Schoneveld L, Othmani A, Abdelkawy H (2021) Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recogn Lett 146:1–7

    Article  Google Scholar 

  32. Sellam A, Azzoune H (2020) Neighborhood min distance descriptor for kinship verification. Multimed Tool Appl 79(29):20,861–20,880

    Article  Google Scholar 

  33. Shadrikov A (2020) Achieving better kinship recognition through better baseline. In: 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020), pp 872–876. IEEE

  34. Wang S, Ding Z, Fu Y (2019) Cross-generation kinship verification with sparse discriminative metric. IEEE Trans Pattern Anal Mach Intell 41 (11):2783–2790. https://doi.org/10.1109/TPAMI.2018.2861871

    Article  Google Scholar 

  35. Wang S, Robinson JP, Fu Y (2017) Kinship verification on families in the wild with marginalized denoising metric learning. In: 12Th IEEE international conference on automatic face & gesture recognition, FG 2017, Washington, DC, USA, May 30 - June 3, 2017, pp 216–221. IEEE Computer Society, DOI https://doi.org/10.1109/FG.2017.35, (to appear in print)

  36. Yan H, Lu J, Deng W, Zhou X (2014) Discriminative multimetric learning for kinship verification. IEEE Trans Inf Forensics Secur 9(7):1169–1178. https://doi.org/10.1109/TIFS.2014.2327757

    Article  Google Scholar 

  37. Zhang K, Huang Y, Song C, Wu H, Wang L (2015) Kinship verification with deep convolutional neural networks. In: Xie X, Jones MW, Tam GKL (eds) Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pp 148.1–148.12. BMVA Press, DOI https://doi.org/10.5244/C.29.148, (to appear in print)

Download references

Author information

Authors and Affiliations

  1. Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine, 94400, France

    Alice Othmani

  2. École d’Ingénieurs Généraliste du Numérique - EFREI Paris, Villejuif, 94800, France

    Duqing Han, Xin Gao & Runpeng Ye

  3. Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, Abu Dhabi, UAE

    Abdenour Hadid

Authors
  1. Alice Othmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Duqing Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Runpeng Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abdenour Hadid
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alice Othmani.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Othmani, A., Han, D., Gao, X. et al. Kinship recognition from faces using deep learning with imbalanced data. Multimed Tools Appl 82, 15859–15874 (2023). https://doi.org/10.1007/s11042-022-14058-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14058-6

Keywords

Search

Navigation