Personalized image quality assessment with Social-Sensed aesthetic preference
Introduction
Aesthetics has been discussed by philosophers for ages. Recently, computer scientists have started to delve into this topic, and opened a new field of research named computational aesthetics. Different from philosophical studies, researchers in this field try to develop computational methods to make applicable aesthetic decisions in a similar fashion as humans can [1]. One key issue in computational aesthetics is to assess image quality from an aesthetic perspective, which is referred to as the problem of image aesthetics assessment. The potential of image aesthetics assessment has been recognized in a broad range of applications, such as image retrieval [2], photo enhancement [3], personal album curation [4], human computer interaction [5], etc.
Image aesthetics assessment is a challenging task due to the lack of a complete set of programmable rules to judge the aesthetic quality. In recent years, there have emerged some large-scale datasets with peer-rated aesthetic scores, which facilitate the development of learning-based methods [6], [7], [8]. Typically, image aesthetics assessment is formulated as a classification or regression problem to distinguish high-aesthetic images from low-aesthetic ones. It is noteworthy that most existing methods are non-personalized, in which an image is assigned a generic aesthetic judgement regardless of the target user. However, as the saying goes, “Beauty is in the eye of the beholder.” Aesthetics is inherently a subjective human experience, and different people may hold different beliefs about the beauty of the same image. Therefore, it is inappropriate to quantify the image aesthetics with merely a generic judgement.
In view of this, personalized image aesthetics assessment would be highly desirable to provide different assessment results regarding the same image for users with different aesthetic preferences. However, a fundamental challenge lies in the lack of understanding of user aesthetic preferences. Recently, several research efforts have been devoted to this issue. For example, Yeh et al. [9] designed an interactive feedback system allowing users to manually adjust feature weights or select preferred photographs for personalized aesthetics ranking. Xu et al. [10] proposed a user-specific aesthetic ranking model with a series of user interactive operations, such as reranking and deletion. Ren et al. [4] collected a labeled aesthetics dataset with raters’ identities, and predicted the residual between the rater-specific ratings and the generic score of an image. Overall, the above studies require users to explicitly express their aesthetic preferences in certain ways, which are time-consuming and labor-intensive.
Nowadays, with the emergence of social media platforms (e.g., Flickr), users not only share and distribute their own photos with others, but also proactively interact with huge volumes of social images. For example, users could mark images as favorite, add tags to images, and organize them into groups. In social psychology, it has proven that human cognition and behavior influence each other [11]. To some extent, favoring expresses the user interests in images, tagging indicates the semantic understanding of images, and grouping identifies the category recognition for images. Therefore, user social behavior can be regarded as the reflection of their cognition to images [12]. Motivated by this, we propose to sense user aesthetic preferences from their favoring behavior on social media platforms. In this way, personalized image aesthetics assessment can be realized without adding any extra burden to users. Besides, unlike prior efforts applicable only to a small select group of users, our work can directly serve the social media user community.
Nevertheless, this idea faces two major problems:
- •
The unreliability of favoring behavior.Generally, the motivation of user favoring behavior is rather complicated, ranging from showing interest in photo contents, strengthening friendship with the owner of the photo, to boosting popularity of the user’s own photos [13]. Favoring behavior may not be evoked by aesthetic stimuli, and users’ favored images can only be regarded as unreliable signals reflecting their aesthetic preferences.
- •
The sparsity of favoring behavior.On social media platforms, numerous users upload and share photos at all times. The large volume of users and images inevitably leads to the sparsity of user-image interactions: most users have only favored a small amount of images [14]. It is quite tricky to capture user aesthetic preferences from such limited resources.
In this paper, in order to alleviate the unreliability problem, we harvest a collection of professional social photos, and consider that user favoring behavior over these images are more likely to identify their tastes in aesthetics. Besides, by means of a generic model trained on large-scale benchmark datasets, an individual’s preference is required to be partially consistent with the common aesthetic standard shared across most users. Given the data sparsity, we gather the favoring behavior of different users together, and model their preferences following the idea of collaborative filtering [15]. The pairwise ranking of images for each user is optimized, so that both favored and non-favored images are jointly leveraged to facilitate the learning process.
The main contributions can be summarized as follows:
- •
A novel perspective is introduced to sense user aesthetic preferences from their favoring behavior on social media platforms, which avoids the cost of requiring users to explicitly express preferences in previous studies.
- •
A new deep neural network architecture is developed for personalized aesthetics modeling, which specifically address the problems of unreliability and sparsity of favoring behavior.
- •
A simulated evaluation scenario is set up to validate our approach on benchmark aesthetics datasets, even though users’ true preferences cannot be directly observed. The results show the potential of our approach for personalized image aesthetics assessment.
The remainder of the paper is structured as follows. Section 2 reviews the related work. Section 3 details our framework for personalized image aesthetics assessment. Section 4 reports the experimental settings and results. Section 5 provides the discussion and limitations of our work. Section 6 concludes this paper.
Section snippets
Related work
In this section, the existing literature on image aesthetics assessment is first reviewed. Then, a brief overview of user profiling and image recommendation is presented, which are two research fields closely related to our work.
Framework
In this section, we introduce a new framework for personalized image aesthetics assessment. The overview of our framework is illustrated in Fig. 1. Firstly, we gather user favoring behavior over professional social photos and consider that they are more likely to identify user aesthetic tastes. Then, a CNN-based generic aesthetics model is developed on benchmark datasets to extract image aesthetic features and capture user shared preferences. Finally, a deep neural network architecture is
Experiments
In this section, a series of experiments were reported to evaluate our approach for personalized image aesthetics assessment. All experiments were carried out on a workstation equipped with a 12-core 3.50 GHz Intel Xeon processor, two Nvidia GTX 1080 GPUs, and 128 GB RAM.
Discussion and limitations
How to understand user aesthetic preferences poses a formidable challenge for personalized image aesthetics assessment. Our goal is to contribute to the progress on this topic by introducing a novel perspective of sensing user aesthetic preferences from their favoring behavior on social media platforms. Towards this purpose, a new framework of personalized image aesthetics assessment is developed, which incorporates many existing advanced techniques, including deep learning and collaborative
Conclusion
In this paper, we have shown the possibility of sensing user aesthetic preferences from their favoring behavior on social media platforms, and avoiding the cost of requiring users to explicitly express preferences. In the modeling, due to the nature of unreliability and sparsity of favoring behavior, we have imposed the requirement of consistency between user personal preference and common aesthetic standard, and followed the idea of collaborative filtering. An effective personalization scheme
Declaration of Competing Interest
We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.
We confirm that we
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant 61701281, Grant 61876098, Grant 61671274, and Grant 61573219, by Shandong Provincial Natural Science Foundation under Grant ZR2017QF009, and by the Fostering Project of Dominant Discipline and Talent Team of Shandong Province Higher Education Institutions.
References (50)
- et al.
The pictures we like are our image: continuous mapping of favorite pictures into self-assessed and attributed personality traits
IEEE Trans. Affect. Comput.
(2017) Defining computational aesthetics, in: proceedings of the 1st eurographics conference on computational aesthetics in graphics
Visual. Imaging Eurograph. Assoc.
(2005)- et al.
Distribution-oriented Aesthetics Assessment for Image Search
Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval
(2017) - L. Mai, H. Jin, F. Liu, Composition-preserving deep photo aesthetics assessment, Proceedings of the IEEE Conference on...
- J. Ren, X. Shen, Z. Lin, R. Mech, D.J. Foran, Personalized image aesthetics, Proceedings of the IEEE Conference on...
- et al.
Rating image aesthetics using deep learning
IEEE Trans Multimed.
(2015) - et al.
Studying aesthetics in photographic images using a computational approach
Proceedings of the 8th European Conference on Computer Vision
(2006) - et al.
Assessing the aesthetic quality of photographs using generic image descriptors
Proceedings of the IEEE International Conference on Computer Vision
(2011) - et al.
Deep aesthetic quality assessment with semantic information
IEEE Trans. Image Process.
(2017) - et al.
Personalized photograph ranking and selection system considering positive and negative user feedback
ACM Trans. Multimed. Comput. Commun. Appl.
(2014)
Usar: an interactive user-specific aesthetic ranking framework for images
Proc. ACM Multimed.
The cognitive impact of past behavior: influences on beliefs, attitudes, and future behavioral decisions
J. Pers Soc. Psychol.
MMALFM: Explainable recommendation by leveraging reviews and images
ACM Trans. Inf. Syst.
Social-Sensed multimedia computing
IEEE Multimed.
Cross-media retrieval with collective deep semantic learning
Multimed. Tools Appl.
Image aesthetic assessment: an experimental survey
IEEE Signal Process Mag.
Content-based photo quality assessment
IEEE Trans Multimed.
Discrete multimodal hashing with canonical views for robust mobile landmark search
IEEE Trans Multimed.
Distribution-oriented aesthetics assessment with semantic-aware hybrid network
IEEE Trans Multimed.
Cited by (25)
Modeling content-attribute preference for personalized image esthetics assessment
2022, Image and Vision ComputingCitation Excerpt :A potential weakness is that it heavily relies on user's online interaction. Cui et al. [9] showed the possibility of modeling user's esthetic preferences from their favorite behaviors on the social networks, which does not need explicit user interaction. However, data from the social network is usually noisy, and there is also a risk of violating user privacy when collecting user data.
The impact of monetary rewards on product sales in referral programs: The role of product image aesthetics
2022, Journal of Business ResearchCitation Excerpt :Despite aesthetics’ subjective nature, recent studies identify some connections between the human aesthetic experience and long-established photographic principles (Zeng, Cao, Zhang, & Bovik, 2020) such as symmetry, contrast, and clarity (Creusen et al., 2010; Grinstein, Hagtvedt, & Kronrod, 2019; Reber et al., 2004). Based on these principles, computer scientists have been able to assess image aesthetics using several computational approaches (Cui et al., 2020; Zeng et al., 2020). We relied on neural image assessment (NIMA) techniques to automatically analyze image aesthetics (Talebi & Milanfar, 2018), an advanced deep convolutional neural network model developed by Google for predicting which images viewers will consider aesthetically attractive.
Individual Contrast Preferences in Natural Images
2024, Journal of ImagingMulti-Level Transitional Contrast Learning for Personalized Image Aesthetics Assessment
2024, IEEE Transactions on MultimediaDeep Temporal State Perception Toward Artificial Cyber-Physical Systems
2023, IEEE Internet of Things JournalIn Pursuit of Beauty: Aesthetic-Aware and Context-Adaptive Photo Selection in Crowdsensing
2023, IEEE Transactions on Knowledge and Data Engineering