Personalized image quality assessment with Social-Sensed aesthetic preference

Abstract

Personalization is emerging as a key research objective for image aesthetics assessment, and how to incorporate personal preferences into aesthetics models is a crucial issue to be solved. Prior studies usually require users to explicitly express their aesthetic preferences in certain ways, which are time-consuming and labor-intensive. In this paper, inspired by the observation that human cognition and behavior influence each other, we propose to sense user aesthetic preferences from their favoring behavior on social media platforms. In this manner, personalized image aesthetics assessment can be realized without adding any extra burden to users. Towards this goal, we gather user favoring behavior over professional social photos and consider both user personal preference and common aesthetic standard to deal with the unreliability of user favoring behavior. Besides, we follow the idea of collaborative filtering and optimize the pairwise ranking between images to alleviate the data sparsity problem. Finally, a deep neural network architecture is developed for personalized aesthetics modeling. A simulated evaluation is carried out on two benchmark aesthetics datasets, even though users’ true preferences cannot be directly observed. The results demonstrate the potential of our approach for personalized image aesthetics assessment.

Introduction

Aesthetics has been discussed by philosophers for ages. Recently, computer scientists have started to delve into this topic, and opened a new field of research named computational aesthetics. Different from philosophical studies, researchers in this field try to develop computational methods to make applicable aesthetic decisions in a similar fashion as humans can [1]. One key issue in computational aesthetics is to assess image quality from an aesthetic perspective, which is referred to as the problem of image aesthetics assessment. The potential of image aesthetics assessment has been recognized in a broad range of applications, such as image retrieval [2], photo enhancement [3], personal album curation [4], human computer interaction [5], etc.

Image aesthetics assessment is a challenging task due to the lack of a complete set of programmable rules to judge the aesthetic quality. In recent years, there have emerged some large-scale datasets with peer-rated aesthetic scores, which facilitate the development of learning-based methods [6], [7], [8]. Typically, image aesthetics assessment is formulated as a classification or regression problem to distinguish high-aesthetic images from low-aesthetic ones. It is noteworthy that most existing methods are non-personalized, in which an image is assigned a generic aesthetic judgement regardless of the target user. However, as the saying goes, “Beauty is in the eye of the beholder.” Aesthetics is inherently a subjective human experience, and different people may hold different beliefs about the beauty of the same image. Therefore, it is inappropriate to quantify the image aesthetics with merely a generic judgement.

In view of this, personalized image aesthetics assessment would be highly desirable to provide different assessment results regarding the same image for users with different aesthetic preferences. However, a fundamental challenge lies in the lack of understanding of user aesthetic preferences. Recently, several research efforts have been devoted to this issue. For example, Yeh et al. [9] designed an interactive feedback system allowing users to manually adjust feature weights or select preferred photographs for personalized aesthetics ranking. Xu et al. [10] proposed a user-specific aesthetic ranking model with a series of user interactive operations, such as reranking and deletion. Ren et al. [4] collected a labeled aesthetics dataset with raters’ identities, and predicted the residual between the rater-specific ratings and the generic score of an image. Overall, the above studies require users to explicitly express their aesthetic preferences in certain ways, which are time-consuming and labor-intensive.

Nowadays, with the emergence of social media platforms (e.g., Flickr), users not only share and distribute their own photos with others, but also proactively interact with huge volumes of social images. For example, users could mark images as favorite, add tags to images, and organize them into groups. In social psychology, it has proven that human cognition and behavior influence each other [11]. To some extent, favoring expresses the user interests in images, tagging indicates the semantic understanding of images, and grouping identifies the category recognition for images. Therefore, user social behavior can be regarded as the reflection of their cognition to images [12]. Motivated by this, we propose to sense user aesthetic preferences from their favoring behavior on social media platforms. In this way, personalized image aesthetics assessment can be realized without adding any extra burden to users. Besides, unlike prior efforts applicable only to a small select group of users, our work can directly serve the social media user community.

Nevertheless, this idea faces two major problems:

• The unreliability of favoring behavior.Generally, the motivation of user favoring behavior is rather complicated, ranging from showing interest in photo contents, strengthening friendship with the owner of the photo, to boosting popularity of the user’s own photos [13]. Favoring behavior may not be evoked by aesthetic stimuli, and users’ favored images can only be regarded as unreliable signals reflecting their aesthetic preferences.

• The sparsity of favoring behavior.On social media platforms, numerous users upload and share photos at all times. The large volume of users and images inevitably leads to the sparsity of user-image interactions: most users have only favored a small amount of images [14]. It is quite tricky to capture user aesthetic preferences from such limited resources.

In this paper, in order to alleviate the unreliability problem, we harvest a collection of professional social photos, and consider that user favoring behavior over these images are more likely to identify their tastes in aesthetics. Besides, by means of a generic model trained on large-scale benchmark datasets, an individual’s preference is required to be partially consistent with the common aesthetic standard shared across most users. Given the data sparsity, we gather the favoring behavior of different users together, and model their preferences following the idea of collaborative filtering [15]. The pairwise ranking of images for each user is optimized, so that both favored and non-favored images are jointly leveraged to facilitate the learning process.

The main contributions can be summarized as follows:

• A novel perspective is introduced to sense user aesthetic preferences from their favoring behavior on social media platforms, which avoids the cost of requiring users to explicitly express preferences in previous studies.

• A new deep neural network architecture is developed for personalized aesthetics modeling, which specifically address the problems of unreliability and sparsity of favoring behavior.

• A simulated evaluation scenario is set up to validate our approach on benchmark aesthetics datasets, even though users’ true preferences cannot be directly observed. The results show the potential of our approach for personalized image aesthetics assessment.

The remainder of the paper is structured as follows. Section 2 reviews the related work. Section 3 details our framework for personalized image aesthetics assessment. Section 4 reports the experimental settings and results. Section 5 provides the discussion and limitations of our work. Section 6 concludes this paper.