Skip to main content
SpringerLink
Log in

Detection and explanation of apparent personality using deep learning: a short review of current approaches and future directions

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Personality describes an individual’s internal attributes, encompassing distinctive thoughts, emotions, and behaviour. Various studies have been undertaken to quantify these personality traits using externally visible cues and measures. Despite the possibility of a disparity between the apparent and real personality traits, researchers have identified various deep learning techniques to measure the apparent personality as part of the ChaLearn Looking at People, ECCV challenge. We provide an overview of the different deep learning models used for apparent personality detection. The primary objective of previous studies was to develop highly accurate prediction models, and subsequently, the focus shifted towards interpreting the output of these techniques. Although explainable AI (XAI) techniques have shown that facial features are essential in predicting personality, the consistency of these results is still a critical factor to consider. Moreover, the majority of studies utilized the ECCV challenge CVPR’17 dataset to predict apparent personality; therefore, there is a requirement to validate these findings through additional datasets. In conclusion, ample scope exists for further research in the domain of personality detection that can yield highly accurate and more interpretable models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

Not applicable

Code availability

Not applicable

References

  1. APA Dictionary of Psychology (2022). https://dictionary.apa.org/

  2. Hakimi S, Hejazi E, Lavasani MG (2011) The relationships between personality traits and students’ academic achievement. Procedia Soc Behav Sci 29:836–845. https://doi.org/10.1016/j.sbspro.2011.11.312

    Article  Google Scholar 

  3. Jensen M (2015) Personality traits, learning and academic achievements. J Educ Learn 4(4):91. https://doi.org/10.5539/jel.v4n4p91

    Article  Google Scholar 

  4. H ÿz (2016) The importance of personality traits in students perceptions of metacognitive awareness. Procedia Soc Behav Sci 232:655–667. https://doi.org/10.1016/j.sbspro.2016.10.090

    Article  Google Scholar 

  5. Juhan MS, Ismail N (2016) Character design towards narrative believability of boboiboy in the malaysian animated feature film boboiboy: the movie (2016). In: Social sciences, paper presented at 2nd international conference on advanced research in economics, social sciences and trade development, p 10

  6. Zammitto V, DiPaola S, Arya A (2008) A methodology for incorporating personality modeling in believable game characters. In: Paper presented at 4th international conference on game reseaarch and development, p 8

  7. Lee KM, Peng W, Jin S, Yan C (2006) Can robots manifest personality?: An empirical test of personality recognition, social responses, and social presence in human-robot interaction. J Commun 56(4):754–772. https://doi.org/10.1111/j.1460-2466.2006.00318.x

    Article  Google Scholar 

  8. Di Napoli C, Ercolano G, Rossi S (2023) Personalized home-care support for the elderly: a field experience with a social robot at home. User Model User Adap Inter 33(2):405–440. https://doi.org/10.1007/s11257-022-09333-y

    Article  Google Scholar 

  9. Reid JA (2011) Crime and personality: personality theory and criminality examined. Inq J 3(01)

  10. Ponce-López V, Chen B, Oliu M, Corneanu C, Clapés A, Guyon I, Baró X, Escalante HJ, Escalera S (2016) ChaLearn LAP 2016: first round challenge on first impressions - dataset and results. In: Hua G, Jégou H (eds.) Computer vision - ECCV 2016 workshops. Lecture notes in computer science, pp. 400–418. Springer, Cham. https://doi.org/10.1007/978-3-319-49409-8_32

  11. Escalante HJ, Guyon I, Escalera S, Jacques J, Madadi M, Baro X, Ayache S, Viegas E, Gucluturk Y, Guclu U, van Gerven MAJ, van Lier R (2017) Design of an explainable machine learning challenge for video interviews. In: 2017 International joint conference on neural networks (IJCNN), pp. 3688–3695. IEEE, Anchorage, AK, USA. https://doi.org/10.1109/IJCNN.2017.7966320

  12. Day DV, Silverman SB (1989) Personality and job performance: evidence of incremental validity. Pers Psychol 42(1):25–36. https://doi.org/10.1111/j.1744-6570.1989.tb01549.x

    Article  Google Scholar 

  13. Barrick MR, Stewart GL, Piotrowski M (2002) Personality and job performance: test of the mediating effects of motivation among sales representatives. J Appl Psychol 87(1):43–51. https://doi.org/10.1037/0021-9010.87.1.43

    Article  Google Scholar 

  14. Hurtz GM, Donovan JJ (2000) Personality and job performance: the big five revisited

  15. Inceoglu I, Warr P (2011) Personality and job engagement. J Pers Psychol 10:177–181. https://doi.org/10.1027/1866-5888/a000045

    Article  Google Scholar 

  16. Dhelim S, Aung N, Bouras MA, Ning H, Cambria E (2022) A survey on personality-aware recommendation systems. Artif Intell Rev 55(3):2409–2454. https://doi.org/10.1007/s10462-021-10063-7

    Article  Google Scholar 

  17. Kirby R, Forlizzi J, Simmons R (2010) Affective social robots. Robot Auton Syst 58(3):322–332. https://doi.org/10.1016/j.robot.2009.09.015

    Article  Google Scholar 

  18. Korotkov D, Hannah TE (2004) The five-factor model of personality: strengths and limitations in predicting health status, sick-role and illness behaviour. Personal Individ Differ 36(1):187–199. https://doi.org/10.1016/S0191-8869(03)00078-3

    Article  Google Scholar 

  19. Hajek A, Bock J-O, König H-H (2017) The role of personality in health care use: results of a population-based longitudinal study in Germany. PLoS ONE 12(7):0181716. https://doi.org/10.1371/journal.pone.0181716

    Article  Google Scholar 

  20. Smith TW, MacKenzie J (2006) Personality and risk of physical illness. Annu Rev Clin Psychol 2(1):435–467. https://doi.org/10.1146/annurev.clinpsy.2.022305.095257

    Article  Google Scholar 

  21. Salazar C, Aguilar J, Monsalve-Pulido J, Montoya E (2021) Affective recommender systems in the educational field. A systematic literature review. Comput Sci Rev 40:100377. https://doi.org/10.1016/j.cosrev.2021.100377

    Article  Google Scholar 

  22. Herzig J, Shmueli-Scheuer M, Sandbank T, Konopnicki D (2017) Neural response generation for customer service based on personality traits. In: Proceedings of the 10th international conference on natural language generation, pp 252–256. Association for Computational Linguistics, Santiago de Compostela, Spain. https://doi.org/10.18653/v1/W17-3541. http://aclweb.org/anthology/W17-3541

  23. Mehta Y, Majumder N, Gelbukh A, Cambria E (2020) Recent trends in deep learning based personality detection. Artif Intell Rev 53(4):2313–2339. https://doi.org/10.1007/s10462-019-09770-z

    Article  Google Scholar 

  24. Birhane A (2022) Automating ambiguity: challenges and pitfalls of artificial intelligence. arXiv. arXiv:2206.04179

  25. Fiske DW (1949) Consistency of the factorial structures of personality ratings from different sources. J Abnormal Soc Psychol 44. https://doi.org/10.1037/h0057198. Place: US Publisher: American Psychological Association

  26. McCrae RR, John OP (1992) An introduction to the five-factor model and its applications. J Pers 60(2):175–215. https://doi.org/10.1111/j.1467-6494.1992.tb00970.x

    Article  Google Scholar 

  27. Costa PT, McCrae RR (1992) The five-factor model of personality and its relevance to personality disorders. J Pers Disord 6(4):343–359. https://doi.org/10.1521/pedi.1992.6.4.343

    Article  Google Scholar 

  28. John OP, Srivastava S (1999) The big-five trait taxonomy: history, measurement, and theoretical perspectives

  29. Amazon Mechanical Turk. https://www.mturk.com/

  30. Junior JCSJ, Lapedriza A, Palmero C, Baro X, Escalera S (2021) Person perception biases exposed: revisiting the first impressions dataset. In: 2021 IEEE winter conference on applications of computer vision workshops (WACVW), pp 13–21. IEEE, Waikola, HI, USA. https://doi.org/10.1109/WACVW52041.2021.00006

  31. Zhao X, Tang Z, Zhang S (2022) Deep personality trait recognition: a survey. Front Psychol 13:839619

    Article  Google Scholar 

  32. Ilmini K, Fernando TGI (2016) Persons’ personality traits recognition using machine learning algorithms and image processing techniques. Adv Comput Sci 5:40–44

    Google Scholar 

  33. Qin R, Gao W, Xu H, Hu Z (2018) Modern physiognomy: an investigation on predicting personality traits and intelligence from the human face. Sci China Inf Sci 61(5):058105. https://doi.org/10.1007/s11432-016-9174-0

    Article  Google Scholar 

  34. Lavater JC (1789) Essays on physiognomy

  35. Chan C-H, Kittler J, Messer K (2007) Multi-scale local binary pattern histograms for face recognition. In: Lee S-W, Li SZ (eds.) Advances in biometrics. Lecture notes in computer science, pp 809–818. Springer, Berlin. https://doi.org/10.1007/978-3-540-74549-5_85

  36. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  37. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vision 42(3):145–175. https://doi.org/10.1023/A:1011139631724

    Article  Google Scholar 

  38. Zhang CL, Zhang H, Wei XS, Wu J (2016) Deep bimodal regression for apparent personality analysis. In: Hua G, Jégou H (eds.) Computer vision - ECCV 2016 workshops. Lecture notes in computer science, pp 311–324. Springer, Cham. https://doi.org/10.1007/978-3-319-49409-8_25

  39. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Procedings of the British machine vision conference 2015, pp 41–14112. British Machine Vision Association, Swansea. https://doi.org/10.5244/C.29.41

  40. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 770–778. IEEE, Las Vegas, NV, USA. https://doi.org/10.1109/CVPR.2016.90

  41. Davis S, Mermelstein P (1980) Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Trans Acoust Speech Signal Process 28(4):357–366. https://doi.org/10.1109/TASSP.1980.1163420

    Article  Google Scholar 

  42. Subramaniam A, Patel V, Mishra A, Balasubramanian P, Mittal A (2016) Bi-modal first impressions recognition using temporally ordered deep audio and stochastic visual features. In: Hua G, Jégou H (eds.) Computer vision - ECCV 2016 workshops. Lecture notes in computer science, pp. 337–348. Springer, Cham. https://doi.org/10.1007/978-3-319-49409-8_27

  43. Giannakopoulos T (2015) pyAudioAnalysis: an open-source python library for audio signal analysis. PLOS ONE 10(12):0144610. https://doi.org/10.1371/journal.pone.0144610

    Article  Google Scholar 

  44. Güçlütürk Y, Güçlü U, van G MAJ, van Lie R (2016) Deep impression: audiovisual deep residual networks for multimodal apparent personality trait recognition. arXiv:1609.05119 [cs]. https://doi.org/10.1007/978-3-319-49409-8_28

  45. Gürpınar F, Kaya H, Salah AA (2016) Combining deep facial and ambient features for first impression estimation. In: Hua G, Jégou H (eds.) Computer vision - ECCV 2016 workshops. Lecture notes in computer science, pp 372–385. Springer, Cham. https://doi.org/10.1007/978-3-319-49409-8_30

  46. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs]

  47. Yang K, Glaser N (2017) Prediction of personality first impressions with deep bimodal LSTM, 10

  48. Barezi EJ, Kampman O, Bertero D, Fung P (2018) Investigating audio, visual, and text fusion methods for end-to- end automatic personality prediction. arXiv:1805.00705 [cs, stat]

  49. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed Representations of words and phrases and their compositionality. arXiv. arXiv:1310.4546 [cs, stat]

  50. Aslan S, Güdükbay U (2019) Multimodal video-based apparent personality recognition using long short-term memory and convolutional neural networks. arXiv:1911.00381 [cs]

  51. He K, Zhang X, Ren S, Sun J (2016) Identity Mappings in deep residual networks. arXiv:1603.05027 [cs]

  52. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503. https://doi.org/10.1109/LSP.2016.2603342

    Article  Google Scholar 

  53. Hershey S, Chaudhuri S, Ellis DPW, Gemmeke JF, Jansen A, Moore RC, Plakal M, Platt D, Saurous RA, Seybold B, Slaney M, Weiss RJ, Wilson K (2017) CNN architectures for large-scale audio classification. arXiv:1609.09430 [cs, stat]. arXiv: 1609.09430

  54. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. arXiv. arXiv:1802.05365 [cs]

  55. Li Y, Wan J, Miao Q, Escalera S, Fang H, Chen H, Qi X, Guo G (2020) CR-net: a deep classification-regression network for multimodal apparent personality analysis. Int J Comput Vision 128(12):2763–2780. https://doi.org/10.1007/s11263-020-01309-y

    Article  Google Scholar 

  56. Mujtaba DF, Mahapatra NR (2021) Multi-task deep neural networks for multimodal personality trait prediction. In: 2021 international conference on computational science and computational intelligence (CSCI), pp 85–91. https://doi.org/10.1109/CSCI54926.2021.00089

  57. Almaev TR, Valstar MF (2013) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 humaine association conference on affective computing and intelligent interaction, pp 356–361. IEEE, Geneva, Switzerland. https://doi.org/10.1109/ACII.2013.65

  58. Li Z, Yang Z, Zhang J, Chen L, He G (2022) Video-based multimodal personality analysis. In: Xu D, Xiao L (eds.) Thirteenth international conference on graphics and image processing (ICGIP 2021), p 156. SPIE, Kunming, China. https://doi.org/10.1117/12.2623646

  59. Song S, Jaiswal S, Sanchez E, Tzimiropoulos G, Shen L, Valstar M (2021) Self-supervised learning of person-specific facial dynamics for automatic personality recognition. IEEE Trans Affect Comput 1–1. https://doi.org/10.1109/TAFFC.2021.3064601

  60. Palmero C, Barquero G, Junior JCSJ, Clapés A, Núñez J, Curto D, Smeureanu S, Selva J, Zhang Z, Saeteros D, Gallardo-Pujol D, Guilera G, Leiva D, Han F, Feng X, He J, Tu WW, Moeslund TB, Guyon I, Escalera S (2022) ChaLearn LAP challenges on self-reported personality recognition and non-verbal behavior forecasting during social dyadic interactions: dataset, design, and results. In: Understanding social behavior in dyadic and small group interactions. Proceedings of Machine Learning Research, vol 173, pp 4–52. PMLR, Canada. ISSN: 2640-3498

  61. Zhang L, Peng S, Winkler S (2022) PersEmoN: a deep network for joint analysis of apparent personality, emotion and their relationship. IEEE Trans Affect Comput 13(1):298–305. https://doi.org/10.1109/TAFFC.2019.2951656

    Article  Google Scholar 

  62. Ventura C, Masip D, Lapedriza A (2017) Interpreting CNN models for apparent personality trait regression. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1705–1713. IEEE, Honolulu, HI, USA. https://doi.org/10.1109/CVPRW.2017.217

  63. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2015) Learning deep features for discriminative localization. arXiv:1512.04150 [cs]. arXiv: 1512.04150

  64. Ekman P, Friesen W, Ancoli S (1980) Facial signs of emotional experience. J Pers Soc Psychol 39:1125–1134

    Article  Google Scholar 

  65. Wei XS, Zhang CL, Zhang H, Wu J (2018) Deep bimodal regression of apparent personality traits from short video sequences. IEEE Trans Affect Comput 9(3):303–315. https://doi.org/10.1109/TAFFC.2017.2762299

    Article  Google Scholar 

  66. Waskom M (2021) Seaborn: statistical data visualization. J Open Source Softw 6(60): 3021. https://doi.org/10.21105/joss.0302

  67. Ilmini W, Fernando T (2023) Explaining the outputs of convolutional neural network-recurrent neural network (CNN-RNN) based apparent personality detection models using the class activation maps. Int J Adv Comput Sci Appl 14(2). https://doi.org/10.14569/IJACSA.2023.0140224

  68. Liao R, Song S, Gunes H (2022) An open-source benchmark of deep learning models for audio-visual apparent and self-reported personality recognition. arXiv. arXiv:2210.09138 [cs]

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Faculty of Graduate Studies, University of Sri Jayewardenepura, Gangodawila, Nugegoda, 10250, Western, Sri Lanka

    WMKS Ilmini

  2. Department of Computer Science, Faculty of Applied Sciences, University of Sri Jayewardenepura, Gangodawila, Nugegoda, 10250, Western, Sri Lanka

    TGI Fernando

  3. Intelligent Research Laboratory, Faculty of Computing, General Sir John Kotelawala Defence University, Ratmalana, Sri Lanka

    WMKS Ilmini

Authors
  1. WMKS Ilmini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. TGI Fernando
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

This manuscript was prepared by WMKS Ilmini under the guidance and supervision of Professor TGI Fernando.

Corresponding author

Correspondence to WMKS Ilmini.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Consent for publication

All co-authors have seen and agreed with the contents of the manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ilmini, W., Fernando, T. Detection and explanation of apparent personality using deep learning: a short review of current approaches and future directions. Computing 106, 275–294 (2024). https://doi.org/10.1007/s00607-023-01221-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-023-01221-6

Keywords

Mathematics Subject Classification

Search

Navigation