Skip to main content
SpringerLink
Log in

TransBLS: transformer combined with broad learning system for facial beauty prediction

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Facial beauty prediction (FBP) is a frontier topic in the fields of machine learning and computer vision, focusing on how to enable computers to judge facial beauty like humans. The existing FBP methods are mainly based on deep neural networks (DNNs). However, DNNs lack global characteristics and only build local dependencies, so FBP still suffers from insufficient supervision information, low accuracy and overfitting. A transformer is a self-attention-based architecture that possesses better global characteristics than DNNs and can build long-term dependencies. Transformers have been widely used to solve some computer vision problems in recent years and have produced better results. In this paper, we propose an adaptive transformer with global and local multihead self-attention for FBP, called GLAFormer. However, GLAFormer does not converge and is prone to overfitting when the training samples are insufficient. The broad learning system (BLS) can accelerate the model convergence process and reduce overfitting. Therefore, we further combine GLAFormer with the BLS to form TransBLS, in which a GLAFormer block is designed as a feature extractor, the features extracted by it are transferred to the BLS for further refining and fitting, and the results are output. Experimental results indicate that TransBLS achieves state-of-the-art FBP performance on several datasets with different scales, better solving the low accuracy and overfitting problems encountered in FBP. It can also be widely applied in pattern recognition and object detection tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4

Similar content being viewed by others

References

  1. Perrett DI, Lee KJ, Penton-Voak I, Rowland D, Yoshikawa S, Burt DM, Henzi SP, Castles DL, Akamatsu S (1998) Effects of sexual dimorphism on facial attractiveness. Nature 394(6696):884–887

    Article  Google Scholar 

  2. Fan YY, Liu S, Li B, Guo Z, Samal A, Wan J, Li SZ (2017) Label distribution-based facial attractiveness computation by deep residual learning. IEEE Trans Multimed 20(8):2196–2208

    Article  Google Scholar 

  3. Chen F, Xiao X, Zhang D (2016) Data-driven facial beauty analysis: prediction, retrieval and manipulation. IEEE Trans Affect Comput 9(2):205–216

    Article  Google Scholar 

  4. Gan J, Jiang K, Tan H, He G (2020) Facial beauty prediction based on lighted deep convolution neural network with feature extraction strengthened. Chin J Electron 29(2):312–321

    Article  Google Scholar 

  5. Zhai Y, Huang Y, Xu Y, Gan J, Cao H, Deng W, Labati RD, Piuri V, Scotti F (2020) Asian female facial beauty prediction using deep neural networks via transfer learning and multi-channel feature fusion. IEEE Access 8:56892–56907

    Article  Google Scholar 

  6. Xu L, Xiang J, Yuan X (2018) Crnet: Classification and regression neural network for facial beauty prediction. In Pacific Rim Conf Multimed 661–671

  7. Lin L, Liang L, Jin L, Chen W (2019) Attribute-Aware Convolutional Neural Networks for Facial Beauty Prediction. The 28th International Joint Conference on Artificial Intelligence (IJCAI) 847–853

  8. Lebedeva I, Guo Y, Ying F (2021) MEBeauty: a multi-ethnic facial beauty dataset in-the-wild. Neural Comput Appl 1–15

  9. Bougourzi F, Dornaika F, Barrena N, Distante C, Taleb-Ahmed A (2022) CNN based facial aesthetics analysis through dynamic robust losses and ensemble regression. Appl Intell 1–18

  10. Lin L, Liang L, Jin L (2019) Regression guided by relative ranking using convolutional neural network (R3CNN) for facial beauty prediction. IEEE Trans Affect Comput 1–14

  11. Wei W, Ho ES, McCay KD, Damaševičius R, Maskeliūnas R, Esposito A (2021) Assessing facial symmetry and attractiveness using augmented reality. Pattern Anal Appl 1–17

  12. Cao K, Choi KN, Jung H, Duan L (2020) Deep learning for facial beauty prediction. Information 11(8):391

    Article  Google Scholar 

  13. Bougourzi F, Dornaika F, Taleb-Ahmed A (2022) Deep learning based face beauty prediction via dynamic robust losses and ensemble regression. Knowl-Based Syst 242:108246

    Article  Google Scholar 

  14. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 5998–6008

  15. Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, Xu Ch, Xu Y, Yang Z, Zhang Y, Tao D (2023) A survey on vision transformer. IEEE Trans Pattern Analy Mach Intell 45(1):87–110

    Article  Google Scholar 

  16. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: Transformers for image recognition at scale. The 9th International Conference on Learning Representations (ICLR) 1–22

  17. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision 10012–10022

  18. Touvron H., Cord M, Douze M, Massa F, Sablayrolles A, Jgou H (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning 10347–10357

  19. Heo YJ, Yeo WH, Kim BG (2022) Deepfake detection algorithm based on improved vision transformer. Appl Intell 1–16

  20. Masood M, Nawaz M, Malik KM, Javed A, Irtaza A, Malik H (2022) Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward. Appl Intell 1–53

  21. Al-Refai R, Nandakumar K (2023) A Unified Model for Face Matching and Presentation Attack Detection Using an Ensemble of Vision Transformer Features. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 662–671

  22. Shao Z, Li F, Zhou Y, Chen H, Zhu H, Yao R (2023) Identity-invariant representation and transformer-style relation for micro-expression recognition. Appl Intell 1–12

  23. Chen CP, Liu Z (2017) Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Trans Neural Netw Learn Syst 29(1):10–24

    Article  MathSciNet  Google Scholar 

  24. Li Y, Zhang T, Chen CP (2021) Enhanced Broad Siamese Network for Facial Emotion Recognition in Human?Robot Interaction. IEEE Trans Artif Intell 2(5):413–423

    Article  Google Scholar 

  25. Li P, Sheng B, Chen CP (2021) Face sketch synthesis using regularized broad learning system. IEEE Trans Neural Netw Learn Syst 5346–5360

  26. Zhai Y, Yu C, Qin C, Zhou W, Ke Q, Gan J, Labati RD, Piuri V, Scotti F (2020) Facial beauty prediction via local feature fusion and broad learning system. IEEE Access 8:218444–218457

    Article  Google Scholar 

  27. Gan J, Xie X, Zhai Y, He G, Mai C, Luo H (2023) Facial beauty prediction fusing transfer learning and broad learning system. Soft Comput 27:13391–13404

  28. Xie D, Liang L, Jin L, Xu J, Li M (2015) SCUT-FBP: A benchmark dataset for facial beauty perception. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics 1821–1826

  29. Gan J, Zhai Y, Wang B (2017) Unconstrained Facial Beauty Prediction Based on Multi-scale K-Means. Chin J Electron 26(3):548–556

    Article  Google Scholar 

  30. Liang L, Lin L, Jin L, Xie D, Li M (2018) SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction. In: 2018 24th International conference on pattern recognition (ICPR) 1598–1603

  31. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision 3730–3738

  32. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition 770–778

  33. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition 2818–2826

  34. Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning, 6105–6114

  35. Zhang L, Zhang D, Sun MM, Chen FM (2017) Facial beauty analysis based on geometric feature: Toward attractiveness assessment application. Expert Syst Appl 82:252–265

    Article  Google Scholar 

  36. Yu Z, Qin Y, Li X, Zhao C, Lei Z, Zhao G (2023) Deep learning for face anti-spoofing: A survey. IEEE Trans Pattern Anal Mach Intell 45(5):5609–5631

    Google Scholar 

  37. Chen S, Chongjian GE, Tong Z, Wang J, Song Y, Wang J, Luo P (2022) AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition. In: Advances in Neural Information Processing Systems, 1-21

  38. Vaswani A, Ramachandran P, Srinivas A, Parmar N, Hechtman B, Shlens J (2021) Scaling local self-attention for parameter efficient visual backbones. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 12894-12904

  39. Gong X, Zhang T, Chen CP, Liu Z (2021) Research review for broad learning system: Algorithms, theory, and applications. IEEE Transactions on Cybernetics 8922–8950

  40. Yang, F. (2018). A CNN-based broad learning system. In: 2018 IEEE 4th International Conference on Computer and Communications (ICCC) 2105-2109

  41. Huang H, Liu Z, Chen CL, Zhang Y (2022) Hyperspectral image classification via active learning and broad learning system. Appl Intell 1–12

  42. Su L, Xiong L, Yang J (2023) Multi-Attn BLS: Multi-head attention mechanism with broad learning system for chaotic time series prediction. Appl Soft Comput 132:109831

    Article  Google Scholar 

  43. Deng L, Xiao M (2023) A New Automatic Hyperparameter Recommendation Approach Under Low-Rank Tensor Completion Framework. IEEE Trans Pattern Anal Mach Intell 45(4):4038–4050

    Google Scholar 

  44. Tolstikhin IO, Houlsby N, Kolesnikov A, Beyer L, Zhai X, Unterthiner T, Yung J, Steiner A, Keysers D, Uszkoreit J, Lucic MX, Dosovitskiy A (2021) Mlp-mixer: An all-mlp architecture for vision. Adv Neural Inf Process Syst 34:24261–24272

    Google Scholar 

  45. Tharwat A (2020) Classification assessment methods. Appl Comput Inf 17(1):168–192

    Google Scholar 

  46. Gan J, Wu B, Zhai Y, He G, Mai C, Bai Z (2022) Self-corrected noise labeling for face beauty prediction. Chinese Journal of Graph Graph 27(08):2487–2495

    Google Scholar 

  47. Gan J, Jiang K, Tan H, He G (2020) Facial beauty prediction based on lighted deep convolution neural network with feature extraction strengthened. Chin J Electron 29(2):312–321

    Article  Google Scholar 

  48. Gan J, Wu B, Zou Q, Zheng Z, Mai C, Zhai Y, He G, Bai Z (2022) Application Research for Fusion Model of Pseudolabel and Cross Network. Comput Intell Neurosci 99866:1–10

    Google Scholar 

  49. Zhai Y, Cao H, Deng W, Gan J, Piuri V, Zeng J (2019) BeautyNet: Joint multiscale CNN and transfer learning method for unconstrained facial beauty prediction. Comput Intell Neurosci 1910624:1–14

    Google Scholar 

  50. Gan J, Wu B, Zou Q, Zheng Z, Mai C, Zhai Y, He G (2022) Two-input dual-task attention network incorporating noisy label correction mechanism for face beauty prediction. Signal Process 38(10):2124–2133

    Google Scholar 

  51. Gan J, Xiang L, Zhai Y, Mai C, He G, Zeng J, Bai Z, Labati RD, Piuri V, Scotti F (2020) 2M BeautyNet: Facial beauty prediction based on multi-task transfer learning. IEEE Access 8:20245–20256

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61771347, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2019A1515010716 and in part by the Basic Research and Applied Basic Research Key Project in General Colleges and Universities of Guangdong Province under Grant 2018KZDXM073.

Author information

Authors and Affiliations

  1. Department of Intelligent Manufacturing, Wuyi University, Canghou, Jiangmen, 529020, Guangdong, China

    Junying Gan, Xiaoshan Xie, Guohui He & Heng Luo

Authors
  1. Junying Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoshan Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guohui He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heng Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junying Gan.

Ethics declarations

Competing interest

All the the authors declare that they have not known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Junying Gan and Xiaoshan Xie are both contributed equally.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gan, J., Xie, X., He, G. et al. TransBLS: transformer combined with broad learning system for facial beauty prediction. Appl Intell 53, 26110–26125 (2023). https://doi.org/10.1007/s10489-023-04931-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04931-8

Keywords

Search

Navigation