Skip to main content
SpringerLink
Log in

Physiognomy: Personality traits prediction by learning

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

Evaluating individuals’ personality traits and intelligence from their faces plays a crucial role in interpersonal relationship and important social events such as elections and court sentences. To assess the possible correlations between personality traits (also measured intelligence) and face images, we first construct a dataset consisting of face photographs, personality measurements, and intelligence measurements. Then, we build an end-to-end convolutional neural network for prediction of personality traits and intelligence to investigate whether self-reported personality traits and intelligence can be predicted reliably from a face image. To our knowledge, it is the first work where deep learning is applied to this problem. Experimental results show the following three points: 1) “Rule-consciousness” and “Tension” can be reliably predicted from face images. 2) It is difficult, if not impossible, to predict intelligence from face images, a finding in accord with previous studies. 3) Convolutional neural network (CNN) features outperform traditional handcrafted features in predicting traits.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. D. McNeill. The Face: A Natural History, New York, USA: Back Bay Books, 2000.

    Google Scholar 

  2. A. Todorov, C. P. Said, A. D. Engell, N. N. Oosterhof. Understanding evaluation of faces on social dimensions. Trends in Cognitive Sciences, vol. 12, no. 12, pp. 455–460, 2008.

    Article  Google Scholar 

  3. C. C. Ballew II, A. Todorov. Predicting political elections from rapid and unreflective face judgments. Proceedings of the National Academy of Sciences of the United States of America, vol. 104, no. 46, pp. 17948–17953, 2007.

    Article  Google Scholar 

  4. A. C. Little, R. P. Burriss, B. C. Jones, S. C. Roberts. Facial appearance affects voting decisions. Evolution and Human Behavior, vol. 28, no. 1, pp. 18–27, 2007.

    Article  Google Scholar 

  5. I. V. Blair, C. M. Judd, K. M. Chapleau. The influence of afrocentric facial features in criminal sentencing. Psychological Science, vol. 15, no. 10, pp. 674–679, 2004.

    Article  Google Scholar 

  6. D. R. Carney, C. R. Colvin, J. A. Hall. A thin slice perspective on the accuracy of first impressions. Journal of Research in Personality, vol. 41, no. 5, pp. 1054–1072, 2007.

    Article  Google Scholar 

  7. R. S. S. Kramer, J. E. King, R. Ward. Identifying personality from the static, nonexpressive face in humans and chimpanzees: Evidence of a shared system for signaling personality. Evolution and Human Behavior, vol. 32, no. 3, pp. 179–185, 2011.

    Article  Google Scholar 

  8. Q. M. Rojas, D. Masip, A. Todorov, J. Vitriä. Automatic point-based facial trait judgments evaluation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, San Francisco, USA, pp. 2715–2720, 2010.

    Google Scholar 

  9. Q. M. Rojas, D. Masip, A. Todorov, J. Vitria. Automatic prediction of facial trait judgments: Appearance vs. structural models. PLoS One, vol. 6, no. 8, Article number e23323, 2011.

    Google Scholar 

  10. K. Wolffhechel, J. Fagertun, U. P. Jacobsen, W. Majewski, A. S. Hemmingsen, C. L. Larsen, S. K. Lorentzen, H. Jarmer. Interpretation of appearance: The effect of facial features on first impressions and personality. PLoS One, vol. 9, no. 9, Article number e107721, 2014.

    Article  Google Scholar 

  11. K. Kleisner, V. Chvátalová, J. Flegr. Perceived intelligence is associated with measured intelligence in men but not women. PLoS One, vol. 9, no. 3, Article number e81237, 2014.

    Article  Google Scholar 

  12. J. Yosinski, J. Clune, Y. Bengio, H. Lipson. How transferable are features in deep neural networks? In Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS, Montréal, Canada, pp. 3320–3328, 2014.

    Google Scholar 

  13. M. S. Long, Y. Cao, J. M. Wang, M. I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, JMLR, Lille, France, 2015.

    Google Scholar 

  14. O. M. Parkhi, A. Vedaldi, A. Zisserman. Deep face recognition. In Proceedings of British Machine Vision Conference, Swansea, UK, vol. 41, pp. 1–12, 2015.

    Google Scholar 

  15. G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, USA, 2007.

    Google Scholar 

  16. Y. Taigman, M. Yang, M. A. Ranzato, L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1701–1708, 2014.

    Google Scholar 

  17. Y. Sun, X. G. Wang, X. O. Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1891–1898, 2014.

    Google Scholar 

  18. Z. Y. Zhu, P. Luo, X. G. Wang, X. O. Tang. Deep learning identity-preserving face space. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp. 113–120, 2013.

    Google Scholar 

  19. Y. Sun, Y. H. Chen, X. G. Wang, X. O. Tang. Deep learning face representation by joint identification-verification. In Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS, Montréal, Canada, pp. 1988–1996, 2014.

    Google Scholar 

  20. Y. Taigman, M. Yang, M. A. Ranzato, L. Wolf. Webscale training for face identification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp. 2746–2754, 2015.

    Google Scholar 

  21. T. Zhang, Q. L. Dong, Z. Y. Hu. Pursuing face identity from view-specific representation to view-invariant representation. In Proceedings of IEEE International Conference on Image Processing, IEEE, Phoenix, USA, pp. 3244–3248, 2016.

    Google Scholar 

  22. B. Zhao, J. S. Feng, X. Wu, S. C. Yan. A survey on deep learning-based fine-grained object classification and semantic segmentation. International Journal of Automation and Computing, vol. 14, no. 2, pp. 1–17, 2017.

    Article  Google Scholar 

  23. N. Kumar, A. C. Berg, P. N. Belhumeur, S. K. Nayar. Attribute and simile classifiers for face verification. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 365–372, 2009.

    Google Scholar 

  24. Y. Taigman, L. Wolf, T. Hassner. Multiple one-shots for utilizing class label information. In Proceedings of British Machine Vision Conference, London, UK, vol. 2, pp. 1–12, 2009.

    Google Scholar 

  25. M. Guillaumin, J. Verbeek, C. Schmid. Is that you? Metric learning approaches for face identification. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 498–505, 2009.

    Google Scholar 

  26. Q. Yin, X. O. Tang, J. Sun. An associate-predict model for face recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Colorado Springs, USA, pp. 497–504, 2011.

    Google Scholar 

  27. C. Huang, S. H. Zhu, K. Yu. Large scale strongly supervised ensemble metric learning, with applications to face verification and retrieval. arXiv:1212.6094, 2012.

    Google Scholar 

  28. D. Chen, X. D. Cao, L. W. Wang, F. Wen, J. Sun. Bayesian face revisited: A joint formulation. In Proceedings of the 12th European Conference on Computer Vision, Florence, Italy, pp. 566–579, 2012.

    Google Scholar 

  29. T. Berg, P. N. Belhumeur. Tom-vs-pete classifiers and identity-preserving alignment for face verification. In Proceedings of British Machine Vision Conference, Guildford, UK, vol. 129, pp. 1–11, 2012.

    Google Scholar 

  30. D. Chen, X. D. Cao, F. Wen, J. Sun. Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Portland, USA, pp. 3025–3032, 2013.

    Google Scholar 

  31. X. D. Cao, D. Wipf, F. Wen, G. Q. Duan, J. Sun. A practical transfer learning algorithm for face verification. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Barcelona, Spain, pp. 3208–3215, 2013.

    Google Scholar 

  32. F. K. Zaman, A. A. Shafie, Y. M. Mustafah. Robust face recognition against expressions and partial occlusions. International Journal of Automation and Computing, vol. 13, no. 4, pp. 319–337, 2016.

    Article  Google Scholar 

  33. N. N. Oosterhof, A. Todorov. The functional basis of face evaluation. Proceedings of the National Academy of Sciences of the United States of America, vol. 105, no. 32, pp. 11087–11092, 2008.

    Article  Google Scholar 

  34. S. Karson. A Guide to the Clinical Use of the 16 pf, Savoy, USA: Institute for Personality and Ability Testing, 1976.

    Google Scholar 

  35. R. M. Kaplan, D. P. Saccuzzo. Psychological Testing: Principles, Applications, and Issues, Boston, USA: Wadsworth Publishing, 2012.

    Google Scholar 

  36. R. Z. Qin, T. Zhang. Shape initialization without ground truth for face alignment. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Shanghai, China, pp. 1278–1282, 2016.

    Google Scholar 

  37. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of ACM International Conference on Multimedia, ACM, Orlando, USA, 2014.

    Google Scholar 

  38. N. Dalal, B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, San Diego, USA, pp. 886–893, 2005.

    Google Scholar 

  39. T. Ahonen, A. Hadid, M. Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 12, pp. 2037–2041, 2006.

    Article  MATH  Google Scholar 

  40. C. J. Liu, H. Wechsler. Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Transactions on Image Processing, vol. 11, no. 4, pp. 467–476, 2002.

    Article  Google Scholar 

  41. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.

    Article  Google Scholar 

  42. A. Oliva, A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, vol. 42, no. 3, pp. 145–175, 2001.

    Article  MATH  Google Scholar 

Download references

Acknowledgement

We are grateful to all the students of Xiamen Institute of Technology in China who participated in this study.

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Ting Zhang, Ri-Zhen Qin, Qiu-Lei Dong, Wei Gao & Zhan-Yi Hu

  2. Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Qiu-Lei Dong & Zhan-Yi Hu

  3. University of Chinese Academy of Sciences, Beijing, 100049, China

    Ting Zhang, Qiu-Lei Dong & Zhan-Yi Hu

  4. Department of Computer Science & Technology, Xiamen Institute of Technology, Xiamen, 361024, China

    Hua-Rong Xu

Authors
  1. Ting Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ri-Zhen Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiu-Lei Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hua-Rong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhan-Yi Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiu-Lei Dong.

Additional information

This work was supported by National Natural Science Foundation of China (Nos. 61333015, 61421004 and 61375042) and Strategic Priority Research Program of the Chinese Academy of Sciences (No.XDB02070002).

Recommended by Associate Editor Jangmyung Lee

Ting Zhang received the B. Sc. degree in communication engineering from Beijing Jiaotong University, China in 2013. She is currently a Ph. D. degree candidate in the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China.

Her research interests include deep learning and face recognition.

Ri-Zhen Qin received the B. Sc. degree in automation from the Xidian University, China in 2013, received the M. Sc. degree from the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China in 2016.

His research interests include machine learning and face recognition.

Qiu-Lei Dong received the B. Sc. degree in automation from the Northeastern University, China in 2003, received the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences, China in 2008. Currently, he is a professor in the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China.

His research interests include motion analysis, 3D computer vision and pattern classification.

Wei Gao received the B. Sc. degree in computational mathematics, the M. Sc. degree in pattern recognition and intelligent system from Shanxi University, China, and the Ph. D. degree in pattern recognition and intelligent system from Institute of Automation, Chinese Academy of Sciences, China in 2002, 2005 and 2008, respectively. Since July 2008, he has joined Robot Vision Group of National Laboratory of Pattern Recognition where he is currently an associate professor.

His research interests include 3D reconstruction from images and SLAM technology.

Hua-Rong Xu received the M. Sc. and the Ph. D. degrees in computer science from Xiamen University, China in 2003 and 2011, respectively. Now he is a professor of Xiamen Institute of Technology, China. He has worked on computer vision and pattern recognition.

His research interests include 3D computer vision and driverless-navigation.

Zhan-Yi Hu received the B. Sc. degree in automation from the North China University of Technology, China in 1985, and received the Ph.D. degree in computer vision from the University of Liege, Belgium, in 1993. Since 1993, he has been with the National Laboratory of Pattern Recognition at Institute of Automation, Chinese Academy of Sciences, China. From 1997 to 1998, he was a visiting scientist with the Chinese University of Hong Kong, China. From 2001 to 2005, he was an executive panel member with the National High-Tech Research and Development Program (863 Program). From 2005 to 2010, he was a member of the Advisory Committee, National Natural Science Foundation of China. He is currently a research professor of computer vision, the deputy editor-in-chief of the Chinese Journal of CAD and CG, and an associate editor of Science China, and Journal of Computer Science and Technology. He was the Organization Committee Co-Chair of the ICCV2005, and the Program Co-Chair of the ACCV2012.

His research interests include biology-inspired vision and large scale 3D reconstruction from images.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, T., Qin, RZ., Dong, QL. et al. Physiognomy: Personality traits prediction by learning. Int. J. Autom. Comput. 14, 386–395 (2017). https://doi.org/10.1007/s11633-017-1085-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-017-1085-8

Keywords

Search

Navigation