Social profiling through image understanding: Personality inference using convolutional neural networks
Introduction
Two directions have shaped the image understanding field of the last 30 years (Liu et al., 2007): the first is the one of the low-level processing, where basic information is extracted from the pixel values in the form of color histograms, frequency responses etc., and used to create a representation in a vectorial space, where tasks of clustering or classification can be carried out (Carson, Thomas, Belongie, Hellerstein, Malik, 1999, Vailaya, Figueiredo, Jain, Zhang, 2001). In the second direction, the semantic content of the image is extracted by means of segmentation, classification and detection approaches, and used for tasks such as content-based indexing and retrieval (Li, Su, Fei-Fei, Xing, 2010, Smeulders, Worring, Santini, Gupta, Jain, 2000).
The advent of Internet, the capability of dealing with big data, and the diffusion of social media, gave rise to a third way of dealing with images (Jin, Wang, Luo, Yu, Han, 2011, Vinciarelli, Pentland, 2015); specifically, images started being associated with people: in facts, images are now digital objects that could be easily uploaded by a certain user into social platforms such as Facebook, Flickr, and the like. Images can be also tagged as “preferred”, highlighting those shots that naturally meet expectations of one in terms of aesthetical preferences and/or semantic content.
Both of these activities (uploading and tagging pictures) indicate a substantial revolution in how images are used: from means to represent visual aspects of reality, where the ownership of the photo is neglected, they have become personal messages, from the sender (the subject which uploads the photos into a social network, or that selects some shots as favorite) to his receiver(s) (the user of the social network that sees the uploaded or the preferred pictures). In this fresh new perspective, uploading or “preferring” images will communicate something, that is, personal messages as the kind of subjects that one may like (cars, landscapes, people) or the life experiences one is going through. But images communicate more than this, and this fact does represent a true revolution in the image understanding field, with a new layer of image interpretation which has started to be unveiled; to explain this new perspective, the sender/receiver communication perspective discussed above becomes invaluable.
In dyadic face-to-face communications, people share their opinions, experiences and impressions of life by using explicit verbal signals (that is, spoken sentences) and non-verbal signals (for example, by how they deliver the sentences, or by assuming bodily expressions) (Vinciarelli, Mohammadi, 2014, Vinciarelli, Pentland, 2015). Many social psychology studies highlight the fundamental importance of both aspects, the verbal content and the non-verbal signals, for the successful exchange of messages. This two-body communication paradigm is modeled by the Brunswick lens, in the field of social psychology (Brunswik, 1956).
Very recently, the Brunswick lens model has been customized for this new kind of communication by images: in this new setting, personality traits have been considered as the social signals sent with the uploaded images, and whose inference is one of the most intriguing challenges. In this respect, the works of Cristani et al. (2013); Segalin et al. (2016) focused on inferring with a regressor the real personality traits of the sender (collected by self-assessed tests), but also those traits that unacquainted people (the assessors) associate with the sender by looking at her images. In particular, Segalin et al. (2016) showed that the assessors’ evaluations were 1) consistently similar, 2) in partial disagreement with the self-assessed evaluations, and 3) more easily predicted by machine learning techniques. In other words, the act of sharing images online may evoke a common psychological response in the receiving crowd, and this can be reasonably predicted by automatic approaches. Thus, it is possible to build a wisdom of the crowds model of personality profiles from collections of images, based on the impressions these may generate on a general hypothetical audience.
A limitation of the approach in Segalin et al. (2016) is that the features used to describe the images are taken from the computational aesthetics (CA) literature; in practice, CA often focuses on designing features that explain how a particular image has been captured, discarding the content of the images. In addition, given the wide spectrum of subjects appearing in database images, standard object recognition and feature extraction techniques might not be sufficient to capture significant dependencies between the pictures and the personality traits of their owner. This leads to the development of more advanced techniques such as feature learning, carried out in this paper by convolutional neural networks.
Computer vision with convolutional neural networks (CNNs) has received much attention in recent years, as it is well suited for processing large amounts of data and providing outstanding performances in classical problems like object (Krizhevsky et al., 2012) and image style (Karayev et al., 2013) recognition. In fact, our approach fine-tunes CNNs pre-trained for image classification with the intention of co-opting their effective representational power to indirectly capture the aesthetic attributes of photographs, with the ultimate goal of predicting the personality traits associated with them. This allows us to discover more entangled attributes and to better generalize the patterns that identify a trait. In practice, whereas CA features are explicitly crafted to reveal information about the style of an image, remaining agnostic w.r.t. the content of the image, CNNs exhibit no such limitation, capturing both the aesthetic patterns in the pictures and their content, unveiling semantic information (for example, capturing possible recurrent objects preferred by a user).
Experiments have been focused on the PsychoFlickr corpus (Segalin et al., 2016): the dataset provides 200 “favored” images from 300 Flickr users for a total of 60,000 images. Additionally, the personality profile of each user is described in terms of the Big Five traits (Rammstedt and John, 2007) extensively used in psychology: Openness to experience (O), Conscientiousness (C), Extraversion (E), Agreeableness (A) and Neuroticism (N). This information is collected both through a self-assessment questionnaire and an independent group of 12 assessors, rating the image sets of each user. This allows the corpus to supply two different evaluation criteria for the same data.
The experimental results show that the proposed method sufficiently captures what characterizes a certain trait: on a quantitative level, it performs around 10% better on attributed traits than on self-assessed ones, with a best accuracy of 68% on attributed Neuroticism; on a qualitative level, ranking the test images by confidence shows a clear distinction of features, patterns and content between low and high values in a given trait. These results also outperform (Segalin et al., 2016) when suitably re-casted from regression to classification. Finally, we also introduce an online application demo that uses our trained classifiers to predict personality traits given a proposed set of pictures liked by a subject.
In the following sections, we first describe some related work in computer vision and computational aesthetics; we then introduce our approach based on processing the PsychoFlickr corpus using convolutional neural networks, followed by a section discussing the results. Finally, we briefly present our demo and provide some concluding remarks.
Section snippets
Related work
The idea that aesthetic values are connected to features goes back at least to Birkoff in the 1930s (Birkhoff, 1933). Hoenig (Hoenig, 2005) in 2005 comprehensively defined computational aesthetics (CA) as a field of study with many emphasis on three important factors: computational methods, the human aesthetic point of view and the need to focus on objective approaches. CA is an inter-disciplinary area at the crossroad between computer vision and pattern recognition (CVPR), psychology, visual
Our approach
The groundbreaking success of the Convolutional Neural Networks (CNNs) in the ILSVRC challenges (Krizhevsky et al., 2012) have clearly demonstrated the aptitude of these classifiers at deconstructing the elements and features contained within photographs. Most importantly, they share some basic primitive components analyzed in the first works of personality inference (which essentially applied standard CA features): color, composition, textural properties, etc. More in the detail, CNNs
Experiments and results
In this section, we first describe a set of baseline experiments where CNNs are used as feature extractors and classification is performed with linear SVMs; we then apply our approach of fine-tuning pre-trained nets and compare these results against the literature and the baseline ones; in addition, we examine a few experiments on the original regression problem in the PsychoFlickr dataset; finally, we analyze the attributes learned by our models. All experiments were performed on a linux pc
Demo
As a matter of proof that our proposed method effectively works, we developed a web interface where a subject can upload an image or a set of images, paste a web address linked to a picture or select an image form a list already stored in the server that he/she likes1. The proposed demo loads the models of the aesthetic preferences related to the attributed traits and classifies the pictures assigning them to the low or high
Conclusions
In this paper, we examine the problem of relating a set of image preferences to personality traits by using a deep learning framework. We cast this recently introduced application problem as a new level of image understanding that enhances the role of images through considerations on the social aspects of contemporary online activities. The role of social platforms like Flickr, Facebook, Instagram, etc., in building online social personas where most activities are shared to a wide audience
Acknowledgment
Dong Seon Cheng was supported by the Hankuk University of Foreign Studies Research Fund of 2015.
References (47)
- et al.
Personality and preference for surreal paintings
Pers. Individual Differ.
(1997) - et al.
A tale of two sites: twitter vs. facebook and the personality predictors of social media usage
Comput. Human Behav.
(2012) - et al.
Rapid: Rating pictorial aesthetics using deep learning
Proceedings of the ACM International Conference on Multimedia
(2014) - et al.
Personality, creativity and aesthetic preference: comparing psychoticism, sensation seeking, schizotypy and openness to experience
Empirical Stud. Arts
(1998) - et al.
Can computers learn from the aesthetic wisdom of the crowd?
KI-Künstliche Intelligenz
(2013) Aesthetic measure
(1933)Perception and the Representative Design of Psychological Experiments
(1956)- et al.
Feature discovery by deep learning for aesthetic analysis of evolved abstract images
Evolutionary and Biologically Inspired Music, Sound, Art and Design
(2015) - et al.
Blobworld: A system for region-based image indexing and retrieval
Visual Information and Information Systems
(1999) - et al.
Finding Image Features Associated with High Aesthetic Value by Machine Learning
(2013)
Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO FFI): Professional Manual
Unveiling the multimedia unconscious: Implicit cognitive processes and multimedia content analysis
Proceedings of the ACM international conference on Multimedia
Semantics and Aesthetics Inference for Image Search: Statistical Learning Approaches. Ph.D. thesis
Studying aesthetics in photographic images using a computational approach
ECCV
High level describable attributes for predicting aesthetics and interestingness
CVPR
Computational aesthetic evaluation: past and future
Computers and Creativity
Predicting personality from twitter
Privacy, Security, Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on
Personality impressions based on facebook profiles.
ICWSM
Defining computational aesthetics
Proceedings of the Eurographics Conference on Computational Aesthetics in Graphics, Visualization and Imaging
Caffe: convolutional architecture for fast feature embedding
arXiv preprint arXiv:1408.5093
Likeminer: a system for mining the power of ‘like’ in social media networks
Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
Cited by (49)
Personality modeling from image aesthetic attribute-aware graph representation learning
2022, Journal of Visual Communication and Image RepresentationCitation Excerpt :To address this issue, an end-to-end weakly supervised dual convolutional network was proposed to simultaneously model users’ five personality traits, leveraging their local attention mechanism on liked images [14]. In addition, Segalin et al. adopted CNN to train a personality model that maps an image to the personality traits of people who like the image [22]. Although advanced progress has been made in existing works, since not all liked images can reveal users’ personality traits (as shown in Fig. 1), it is unreasonable to directly take users’ personality traits as the supervised labels of all their liked images for model learning.
Fuzzy and genetic algorithm based approach for classification of personality traits oriented social media images
2022, Knowledge-Based SystemsCitation Excerpt :This is because images uploaded on social media are associated with the personality traits of the people that interact with them. Furthermore, a person’s personality traits reflect on their uploading and receiving images [4]. This is understandable because people use social media networks to share their daily activities, expressing emotions, sharing happiness, etc.
Multimodal assessment of apparent personality using feature attention and error consistency constraint
2021, Image and Vision ComputingCitation Excerpt :These studies indicate that there is a strong correlation between users' behavior on social networks and their personality [32]. Additionally, the exploitation of images and words used in public profiles in social networks is a way of obtaining an effective personality trait model, as shown in [33,34]. There are methods for recognition based on combinations of speaking style and body movements.
Psychological targeting in the age of Big Data
2021, Measuring and Modeling Persons and Situations