Skip to main content
Springer Nature Link
Log in

Cross-database facial expression recognition based on hybrid improved unsupervised domain adaptation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Since the labeled wild facial expression database is relatively rare, the existing Facial Expression Recognition (FER) models based on machine learning can only be trained with a relatively limited number of samples and whether the trained FER model can have satisfactory recognition performance is a challenge. In this paper, the facial expression database from the Laboratory Environment (LE) is used as the source domain, and the facial expression database from the wild is used as the target domain. Based on these two different databases, a hybrid improved unsupervised Cross-Domain Adaptation (CDA) approach is proposed, which can not only match the data distribution between different databases, but also maximize the correlation of data between different databases, and also maximize data separability on the source database. In the proposed CDA approach, the objective functions of the two improved techniques and those of traditional CDA are to achieve the simultaneous optimization of the three objective functions. After that, the proposed CDA approach was used for Cross-domain FER (CFER) task. To confirm the effectiveness of the proposed CFER model, some experiments are implemented on four cross-database pairs. The comparison and analysis of experimental results show that, compared with other existing CFER models, the proposed CFER model can realize the reuse of LE facial expression data and achieve better recognition performance for wild facial expression data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://cs.anu.edu.au/few/emotiw2015.html

  2. https://github.com/duanyq14/CA-LBFL

References

  1. Aghamaleki JA, Chenarlogh VA (2019) Multi-stream CNN for facial expression recognition in limited training data. Multimed Tools Appl 78:22861–22882

    Article  Google Scholar 

  2. Alphonse AS, Starvin MS (2019) A novel maximum and minimum response-based Gabor (MMRG) feature extraction method for facial expression recognition. Multimed Tools Appl 78:23369–23397

    Article  Google Scholar 

  3. Bejiga MB, Melgani F (2018) GAN-based domain adaptation for object classification. IEEE International Geoscience and Remote Sensing Symposium, 22–27 July 2018, Valencia, Spain, pp 1264–1267. https://doi.org/10.1109/IGARSS.2018.8518649

  4. Ben X, Jia X, Yan R, Zhang X, Meng W (2018) Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn Lett 107:50–58

    Article  Google Scholar 

  5. Beygelzimer A, Langford J, Ravikumar P (2007) Multiclass classification with filter trees. Technical report, 2 June 2007, Citeseer. http://mi.eng.cam.ac.uk/~mjfg/local/Projects/filter_tree.pdf

  6. Chen W, Hu H (2020) Unsupervised domain adaptation via discriminative classes-center feature learning in adversarial network. Neural Process Lett 52:467–483

    Article  Google Scholar 

  7. Chen C, Jiang BY, Cheng ZW et al (2019) Joint domain matching and classification for cross-domain adaptation via ELM. Neurocomputing 349:314–325

    Article  Google Scholar 

  8. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1:886–893

    Google Scholar 

  9. Daume H III, Marcu D (2006) Domain adaptation for statistical classifiers. J Artif Intell Res 26(1):101–126

    Article  MathSciNet  MATH  Google Scholar 

  10. Dhall A, Goecke R, Lucey S et al (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. IEEE International Conference on Computer Vision Workshops, 6–13 November 2011, Barcelona, Spain, pp 2106–2112. https://doi.org/10.1109/ICCVW.2011.6130508

  11. Duan Y, Lu J, Feng J, Zhou J (2018) Context-aware local binary feature learning for face recognition. IEEE Trans Pattern Anal Mach Intell 40(5):1139–1153

    Article  Google Scholar 

  12. Francisco EC, Thelmo P (2019) D a, José E B M. facial expression recognition: a cross-database evaluation of features and classifiers. Journal of Intelligent Computing 10(1):34–45

    Article  Google Scholar 

  13. Gholenji E, Tahmoresnezhad J (2020) Joint discriminative subspace and distribution adaptation for unsupervised domain adaptation. Appl Intell 50:2050–2066

    Article  Google Scholar 

  14. He Z, Yang B, Chen C, Mu Q, Li Z (2020) CLDA: an adversarial unsupervised domain adaptation method with classifier-level adaptation. Multimed Tools Appl 79:33973–33991

    Article  Google Scholar 

  15. Herath S, Harandi M, Porikli F (2017) Learning an invariant Hilbert space for domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, 21–26 July 2017, Honolulu, Hawaii, pp 3845–3854. https://doi.org/10.1109/CVPR.2017.421

  16. Huang FC, Huang SY, Ker JW, Chen YC (2012) High-performance SIFT hardware accelerator for real-time image feature extraction. IEEE Transactions on Circuits & Systems for Video Technology 22(3):340–351

    Article  Google Scholar 

  17. Huang X, Wang SJ, Liu X, Zhao G, Feng X, Pietikainen M (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47

    Article  Google Scholar 

  18. Islam MN, Loo CK (2014) Geometric feature-based facial emotion recognition using two-stage fuzzy reasoning model. In: Loo, C.K., Yap, K.S., Wong, K.W., Teoh, A., Huang, K. (eds) Neural Information Processing. ICONIP 2014. Lecture Notes in Computer Science, vol 8835. Springer, Cham. https://doi.org/10.1007/978-3-319-12640-1_42

  19. Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239

    Article  Google Scholar 

  20. Jin C (2021) Cross-project software defect prediction based on domain adaptation learning and optimization. Expert Syst Appl 171:114637

    Article  Google Scholar 

  21. Kim BK, Roh JY, Dong SY et al (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. Journal on Multimodal User Interfaces 10(2):173–189

    Article  Google Scholar 

  22. Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, van Knippenberg A (2010) Presentation and validation of the Radboud faces database. Cognit Emot 24(8):1377–1388

    Article  Google Scholar 

  23. Li J, Lu K, Huang Z, Zhu L, Shen HT (2018) Transfer independently together: a generalized framework for domain adaptation. IEEE Transactions on Cybernetics 49(6):2144–2155

    Article  Google Scholar 

  24. Liu N, Zhang B, Zong Y et al (2018) Super wide regression network for unsupervised cross-database facial expression recognition. 43th IEEE International Conference on Acoustics, Speech and Signal Processing, 15–20 April 2018, Calgary, Canada, pp 1897–1901. https://doi.org/10.1109/ICASSP.2018.8461322

  25. Lo SK (2008) The nonverbal communication functions of emoticons in computer-mediated communication. Cyberpsychology & Behavior 11(5):595–597

    Article  Google Scholar 

  26. Long M, Wang J, Ding G et al (2013) Transfer feature learning with joint distribution adaptation. IEEE International Conference on Computer Vision. 1–8 December 2013, Sydney, Australia, pp 2200–2207. https://doi.org/10.1109/ICCV.2013.274

  27. Luo LK, Chen LM, Hu SQ et al (2020) Discriminative and geometry aware unsupervised domain adaptation. IEEE Transactions on Cybernetics 50(9):3914–3927

    Article  Google Scholar 

  28. Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  29. Miao Y, Araujo R, Kamel M et al (2012) Cross-domain facial expression recognition using supervised kernel mean matching. 11th International Conference on Machine Learning and Applications, pp 12–15 December 2012, Boca Raton, USA, 326–332. https://doi.org/10.1109/ICMLA.2012.178

  30. Mollahosseini A, Hasani B, Mahoor MH (2019) AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31

    Article  Google Scholar 

  31. Nguyen V, Le T, Vel O et al (2020) Dual-component deep domain adaptation: a new approach for cross project software vulnerability detection. In: Lauw H, Wong RW, Ntoulas A, Lim EP, Ng SK, Pan S (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2020. Lecture Notes in Computer Science, vol 12084. Springer, Cham. https://doi.org/10.1007/978-3-030-47426-3_54

  32. Ni TG, Zhang C, Gu XQ (2021) Transfer model collaborating metric learning and dictionary learning for cross-domain facial expression recognition. IEEE Transactions on Computational Social Systems 8(5):1213–1222

    Article  Google Scholar 

  33. Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw 22(2):199–210

    Article  Google Scholar 

  34. Rajan S, Chenniappan P, Devaraj S, Madian N (2020) Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process 14(7):1373–1381

    Article  Google Scholar 

  35. Richhariya B, Gupta D (March 2019) Facial expression recognition using iterative universum twin support vector machine. Appl Soft Comput 76:53–67

    Article  Google Scholar 

  36. Rostami M, Berahmand K, Nasiri E, Forouzandeh S (2021) Review of swarm intelligence-based feature selection methods. Eng Appl Artif Intell 100:104210

    Article  Google Scholar 

  37. Rostami M, Forouzandeh S, Berahmand K, Soltani M, Shahsavari M, Oussalah M (2022) Gene selection for microarray data classification via multi-objective graph theoretic-based method. Artif Intell Med 123:102228

    Article  Google Scholar 

  38. Sanodiya RK, Mathew J (2019) A framework for semi-supervised metric transfer learning on manifolds. Knowl-Based Syst 176:1–14

    Article  Google Scholar 

  39. Sanodiya RK, Mathew J, Saha S, Thalakottur MD (2019) A new transfer learning algorithm in semi-supervised setting. IEEE Access 7:42956–42967

    Article  Google Scholar 

  40. Seyed MH, Mohammad R (2020) Cross-domain recommender system using generalized canonical correlation analysis. Knowl Inf Syst 62(12):4625–4651

    Article  Google Scholar 

  41. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  42. Sharma M, Jalal AS, Khan A (2019) Emotion recognition using facial expression by fusing key points descriptor and texture features. Multimed Tools Appl 78(12):16195–16219

    Article  Google Scholar 

  43. Shen C, Guo Y H (2018) Unsupervised heterogeneous domain adaptation with sparse feature transformation. 10th Asian Conference on Machine Learning, 14–16 November 2018, Beijing, China, pp 375–390

  44. Shinnou H, Onodera Y, Sasaki M et al (2015) Active learning to remove source instances for domain adaptation for word sense disambiguation. In: Hasida, K., Purwarianti, A. (eds) Computational Linguistics. PACLING 2015. Communications in Computer and Information Science, vol 593. Springer, Singapore. https://doi.org/10.1007/978-981-10-0515-2_7

  45. Sun W, Yuan YX (2006) Theory of constrained optimization. In: numerical optimization. In: Optimization Theory and Methods. Springer Optimization and Its Applications, vol 1. Springer, Boston. https://doi.org/10.1007/0-387-24976-1_8

  46. Tahmoresnezhad J, Hashemi S (2016) Visual domain adaptation via transfer feature learning. Knowledge & Information Systems 50(2):1–21

    Google Scholar 

  47. Usman M, Latif S, Qadir J (2017) Using deep autoencoders for facial expression recognition. 2017 13th International Conference on Emerging Technologies (ICET), Islamabad, Pakistan, 27-28 December 2017, pp 1–6. https://doi.org/10.1109/ICET.2017.8281753

  48. Wang M, Deng W (2018) Deep visual domain adaptation: a survey. Neurocomputing 312:135–153

    Article  Google Scholar 

  49. Wang YY, Nie LL, Li Y et al (2020) Soft large margin clustering for unsupervised domain adaptation. Knowl-Based Syst 192:105344

    Article  Google Scholar 

  50. Xie Y, Chen T S, Pu T et al (2020) Adversarial graph representation adaptation for cross-domain facial expression recognition. MM '20: Proceedings of the 28th ACM International Conference on Multimedia October 2020, pp 1255–1264. https://doi.org/10.1145/3394171.3413822

  51. Xu Y, Pan SJ, Xiong H, Wu Q, Luo R, Min H, Song H (2017) A unified framework for metric transfer learning. IEEE Trans Knowl Data Eng 29(6):1158–1171

    Article  Google Scholar 

  52. Xu XM, He H, Zhang HD et al (2020) Unsupervised domain adaptation via importance sampling. IEEE Transactions on Circuits and Systems for Video Technology 30(12):4688–4699

    Article  Google Scholar 

  53. Yan KY, Zheng WM, Zhang T et al (2019) Cross-domain facial expression recognition based on transductive deep transfer learning. IEEE Access 7:108906–108915

    Article  Google Scholar 

  54. Yang Z, Liu G, Xie X, Cai Q (2020) Efficient dynamic domain adaptation on deep CNN. Multimed Tools Appl 79:33853–33873

    Article  Google Scholar 

  55. Yang Y, Vuksanovic B, Ma H (2020) Effects of region features on the accuracy of cross-database facial expression recognition. In: Proceedings of the 12th International Conference on Agents and Artificial Intelligence - vol 2: ICAART, pp 610–617. https://doi.org/10.5220/0008966306100617

  56. Zellinger W, Moser BA, Grubinger T, Lughofer E, Natschläger T, Saminger-Platz S (May 2019) Robust unsupervised domain adaptation for neural networks via moment alignment. Inf Sci 483:174–191

    Article  MathSciNet  MATH  Google Scholar 

  57. Zellinger W, Moser BA, Saminger-P S (2021) On generalization in moment-based domain adaptation. Ann Math Artif Intell 89(3–4):333–369

    Article  MathSciNet  MATH  Google Scholar 

  58. Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12):2528–2536

    Article  Google Scholar 

  59. Zhang J, Li W Q, Ogunbona P (2017) Joint geometrical and statistical alignment for visual domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5150–5158. https://doi.org/10.1109/CVPR.2017.547

  60. Zhao C, Wang S, Li D (March 2020) Multi-source domain adaptation with joint learning for cross-domain sentiment classification. Knowl-Based Syst 191:105254

    Article  Google Scholar 

  61. Zheng W, Zong Y, Zhou X, Xin M (2018) Cross-domain color facial expression recognition using transductive transfer subspace learning. IEEE Trans Affect Comput 9(1):21–37

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer, Central China Normal University, Wuhan, 430079, People’s Republic of China

    Cong Jin

Authors
  1. Cong Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cong Jin.

Ethics declarations

Conflict of interest

The author declares that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, C. Cross-database facial expression recognition based on hybrid improved unsupervised domain adaptation. Multimed Tools Appl 82, 1105–1129 (2023). https://doi.org/10.1007/s11042-022-13311-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13311-2

Keywords

Search

Navigation