Skip to main content
SpringerLink
Log in

Personalized broad learning system for facial expression

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Understanding human emotions through facial expressions is key enabling technology for interactive robots. Most approaches of facial expression recognition are designed for the average user. It is difficult for them to maintain high accuracy for special users with different cultural backgrounds and personalities, which limits their application in real world scenarios. Personalized classifier is a feasible solution, but it needs to be retrained for new users outside the training set. In this paper we present a framework for personalizing facial expression recognition which does not require re-training models after entering new data. Personalized incremental updating mechanism is achieved by designing a novel broad learning system. Specifically, we propose a transfer learning model based on emotional information entropy as the mapping feature layer to ensure the accuracy of mapping under the condition of small sample size. Then, the weights of our proposed model can be updated by multi-layer singular value decomposition method if incremental data is entered. We exhibit the superiority of our approach in multiple facial expression datasets. Experimental results show that our method has higher accuracy and generalization ability with previous personalization techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Chang H, Han J, Zhong C, Snijders AM, Mao JH (2018) Unsupervised transfer learning via multi-scale convolutional sparse coding for biomedical applications. IEEE Trans Pattern Anal Mach Intell 40:1182–1194. https://doi.org/10.1109/TPAMI.2017.2656884

    Article  Google Scholar 

  2. Chen CLP, Liu Z (2018) Broad learning system: an effective and efficient incremental learning system without the need for deep architecture. IEEE Trans Neural Netw 29:10–24. https://doi.org/10.1109/TNNLS.2017.2716952

    Article  MathSciNet  Google Scholar 

  3. Chen Z, Jiang B, Tang J et al (2017) Image set representation and classification with attributed covariate-relation graph model and graph sparse representation classification. Neurocomputing 226:262–268. https://doi.org/10.1016/j.neucom.2016.12.004

    Article  Google Scholar 

  4. Cho D, Tai YW, Kweon IS (2019) Deep convolutional neural network for natural image matting using initial alpha mattes. IEEE Trans Image Process 28:1054–1067. https://doi.org/10.1109/TIP.2018.2872925

    Article  MathSciNet  Google Scholar 

  5. Chu WS, De la Torre F, Cohn JF (2017) Selective transfer machine for personalized facial expression analysis. IEEE Trans Pattern Anal Mach Intell 39:529–545. https://doi.org/10.1109/TPAMI.2016.2547397

    Article  Google Scholar 

  6. Feng S, Chen CP (2018) Fuzzy broad learning system: a novel neuro-fuzzy model for regression and classification. IEEE Trans Cybernet 99:1–11. https://doi.org/10.1109/TCYB.2018.2857815

    Article  Google Scholar 

  7. Feydy J, Séjourné T, Vialard FX, et al (2018) Interpolating between optimal transport and MMD using Sinkhorn divergences. arXiv preprint 2018, arXiv.: 1810.08278

  8. Gando G, Yamada T, Sato H, Oyama S, Kurihara M (2016) Fine-tuning deep convolutional neural networks for distinguishing illustrations from photographs. Expert Syst Appl 66:295–301. https://doi.org/10.1016/j.eswa.2016.08.057

    Article  Google Scholar 

  9. Goyani MM, Patel N (2017) Recognition of facial expressions using local mean binary pattern. ELCVIA 16:54–67. https://doi.org/10.5565/rev/elcvia.1058

    Article  Google Scholar 

  10. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science. 313:504–507. https://doi.org/10.1126/science.1127647

    Article  MathSciNet  MATH  Google Scholar 

  11. Hong M, Luo ZQ (2017) On the linear convergence of the alternating direction method of multipliers. Math Program 162(1-2):165–199. https://doi.org/10.1007/s10107-016-1034-2

    Article  MathSciNet  MATH  Google Scholar 

  12. Hong H, Neven H, Von der Malsburg C (1998) Online facial expression recognition based on personalized galleries. Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition. Nara, Japan, p 354-359

  13. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern B 42:513–529. https://doi.org/10.1109/TSMCB.2011.2168604

    Article  Google Scholar 

  14. Jia S, Cristianini N (2015) Learning to classify gender from four million images. Pattern Recogn Lett 58:35–41. https://doi.org/10.1016/j.patrec.2015.02.006

    Article  Google Scholar 

  15. Jiang R, Ho AT, Cheheb I, Al-Maadeed N, Al-Maadeed S, Bouridane A (2017) Emotion recognition from scrambled facial images via many graph embedding. Pattern Recogn 67:245–251. https://doi.org/10.1016/j.patcog.2017.02.003

    Article  Google Scholar 

  16. Kokkinos Y, Margaritis KG (2018) Managing the computational cost of model selection and cross-validation in extreme learning machines via Cholesky, SVD, QR and eigen decompositions. Neurocomputing. 295:29–45. https://doi.org/10.1016/j.neucom.2018.01.005

    Article  Google Scholar 

  17. Kong Y, Wang XS, Cheng YH, Chen CL (2018) Hyperspectral imagery classification based on semi-supervised broad learning system. Remote Sens 10:685–698. https://doi.org/10.3390/rs10050685

    Article  Google Scholar 

  18. Kow KW, Wong YW, Rajkumar RK, Rajkumar RK, Isa D (2016) Incremental unsupervised learning algorithm for power fluctuation event detection in PV grid-tied systems. In: Proceedings of the 9th International Conference on Robotic, Vision, Signal Processing and Power Applications, Singapore, p 673-679.

  19. Lee J, Kim H, Lee J, Yoon S (2017) Transfer learning for deep learning on graph-structured data. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. San Francisco, California, USA, p 2154–2160

  20. Li C, de Rijke M (2018) Incremental sparse Bayesian ordinal regression. Neural Netw 106:294–302. https://doi.org/10.1016/j.neunet.2018.07.015

    Article  MATH  Google Scholar 

  21. Lu J, Behbood V, Hao P, Zuo H, Xue S, Zhang G (2015) Transfer learning using computational intelligence: a survey. Knowl-Based Syst 80:14–23. https://doi.org/10.1016/j.knosys.2015.01.010

    Article  Google Scholar 

  22. Martinez DL, Rudovic O, Picard R (2017) Personalized automatic estimation of self-reported pain intensity from facial expressions. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA, p 2318-2327

  23. Monteiro JC, Cardoso JS (2015) A cognitively-motivated framework for partial face recognition in unconstrained scenarios. Sensors. 15:1903–1924. https://doi.org/10.3390/s150101903

    Article  Google Scholar 

  24. Owusu E, Zhan Y, Mao QR (2014) An SVM-AdaBoost facial expression recognition system. Appl Intell 40:536–545. https://doi.org/10.1007/s10489-013-0478-9

    Article  Google Scholar 

  25. Pratama M, Lu J, Lughofer E, Zhang GQ, Er MJ (2017) An incremental learning of concept drifts using evolving type-2 recurrent fuzzy neural networks. IEEE Trans Fuzzy Syst 25:1175–1192. https://doi.org/10.1109/TFUZZ.2016.2599855

    Article  Google Scholar 

  26. Rebuffi SA, Bilen H, Vedaldi A (2018) Efficient parametrization of multi-domain deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah., USA, p 8119-8127

  27. Ruiz A, Rudovic O, Binefa X et al (2018) Multi-instance dynamic ordinal random fields for weakly supervised facial behavior analysis. IEEE Trans Image Process 27:3969–3982. https://doi.org/10.1109/TIP.2018.2830189

    Article  MathSciNet  MATH  Google Scholar 

  28. Russakovsky O, Deng J, Su H et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  29. Siddiqi MH, Ali R, Khan AM et al (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24:1386–1398. https://doi.org/10.1109/TIP.2015.2405346

    Article  MathSciNet  MATH  Google Scholar 

  30. Tao D, Guo Y, Li Y et al (2018) Tensor rank preserving discriminant analysis for facial recognition. IEEE Trans Image Process 27:325–334. https://doi.org/10.1109/TIP.2017.2762588

    Article  MathSciNet  MATH  Google Scholar 

  31. Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Hawaii, USA, 2017, pp. 1-4

  32. Uçar A, Demir Y, Güzeliș C (2016) A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering. Neural Comput & Applic 27:131–142. https://doi.org/10.1007/s00521-014-1569-1

    Article  Google Scholar 

  33. Wang JFJ, Tarn DD (2018) Are two heads better than one?–intellectual capital, learning and knowledge sharing in a dyadic interdisciplinary relationship. J Knowl Manag 22:1379–1407. https://doi.org/10.1108/JKM-04-2017-0145

    Article  Google Scholar 

  34. Xie L, Li G, Peng L, Chen Q, Tan Y, Xiao M (2017) Band selection algorithm based on information entropy for hyperspectral image classification. J Appl Remote Sens 11:026018

    Article  Google Scholar 

  35. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9:1–8. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  36. Yang S, Rudovic O, Pavlovic V, et al (2014) Personalized modeling of facial action unit intensity. International Symposium on Visual Computing. Cham, Germany, 2014, p 269-281. https://doi.org/10.1007/978-3-319-14364-4_26

  37. Zen G, Porzi L, Sangineto E, Ricci E, Sebe N (2016) Learning personalized models for facial expression analysis and gesture recognition. IEEE Trans Multimedia 18:775–788. https://doi.org/10.1109/TMM.2016.2523421

    Article  Google Scholar 

  38. Zhang Z, Luo P, Loy CC et al (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126:550–569. https://doi.org/10.1007/s11263-017-1055-1

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work is supported by the National Key Research and Development Program of China (No. 2017YFB1302200), National Natural Science Foundation of China (No.61672093), Advanced Innovation Center for Intelligent Robots and Systems Open Research Project (No.2018IRS01), National Natural Science Foundation of China (Key Project No.61432004).

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China

    Jing Han & Lun Xie

  2. Peking University Institute of Mental Health, Beijing, 100191, China

    Jing Liu & Xue Li

Authors
  1. Jing Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lun Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xue Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lun Xie.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, J., Xie, L., Liu, J. et al. Personalized broad learning system for facial expression. Multimed Tools Appl 79, 16627–16644 (2020). https://doi.org/10.1007/s11042-019-07979-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07979-2

Keywords

Search

Navigation