skip to main content
10.1145/3581783.3611942acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Personalized Image Aesthetics Assessment with Attribute-guided Fine-grained Feature Representation

Published: 27 October 2023 Publication History

Abstract

Personalized image aesthetics assessment (PIAA) has gained increasing attention from researchers due to its ability to measure individual users' specific aesthetic experiences. However, most existing PIAA methods rely on holistic features or simplistic coding to characterize users' aesthetic preferences for images, and we believe that more rich explicit features are needed in modeling PIAA. Consequently, we propose an attribute-guided fine-grained feature-aware personalized image aesthetics assessment method, which can fully capture fine-grained features from multiple attributes to represent users' aesthetic preferences for images. To achieve this, we first build a fine-grained feature extraction (FFE) module to obtain the refined local features of image attributes to compensate for holistic features. The FFE module is then used to generate user-level features, which are combined with the image-level features to obtain user-preferred fine-grained feature representations. By training extensive users' PIAA tasks, the aesthetic distribution of most users can be transferred to the personalized scores of individual users. To enable our proposed model to learn more generalizable aesthetics among individual users, we incorporate the degree of dispersion between users' personalized scores and image aesthetic distribution as a coefficient in the loss function during model training. Experimental results on several PIAA databases show that our method outperforms existing mainstream PIAA methods, and can effectively infer users' personalized aesthetics of images.

Supplemental Material

MP4 File
This is a video presentation of the paper entitled "Personalized Image Aesthetics Assessment with Attribute-guided Fine-grained Feature Representation" at the ACM Multimedia 2023 conference. This video briefly introduces some background information and our proposed method. For more detailed information, please refer to our paper.

References

[1]
Simone Bianco, Claudio Cusano, Flavio Piccoli, and Raimondo Schettini. 2020. Personalized Image Enhancement Using Neural Spline Color Transforms. IEEE Trans. Image Process. 29 (2020), 6223--6236.
[2]
Luigi Celona, Marco Leonardi, Paolo Napoletano, and Alessandro Rozza. 2022. Composition and Style Attributes Guided Image Aesthetic Assessment. IEEE Trans. Image Process. 31 (2022), 5009--5024.
[3]
Chaoran Cui, Huihui Liu, Tao Lian, Liqiang Nie, Lei Zhu, and Yilong Yin. 2019. Distribution-Oriented Aesthetics Assessment With Semantic-Aware Hybrid Network. IEEE Trans. Multimedia 21, 5 (2019), 1209--1220.
[4]
Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2017. Image Aesthetic Assessment: An Experimental Survey. IEEE Signal Process. Mag. 34, 4 (2017), 80--106.
[5]
Hossein Talebi Esfandarani and Peyman Milanfar. 2018. NIMA: Neural Image Assessment. IEEE Trans. on Image Process. 27, 8 (2018), 3998--4011.
[6]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 770--778.
[7]
Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. 2022. Rethinking Image Aesthetics Assessment: Models, Datasets and Benchmarks. In Proc. Int. Joint Conf. Artif. Intell. 942--948.
[8]
Chaoyi Hong, Shuaiyuan Du, Ke Xian, Hao Lu, Zhiguo Cao, and Weicai Zhong. 2021. Composing Photos Like a Photographer. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 7053--7062.
[9]
Richang Hong, Luming Zhang, and Dacheng Tao. 2016. Unified Photo Enhancement by Discovering Aesthetic Communities From Flickr. IEEE Trans. Image Process. 25, 3 (2016), 1124--1135.
[10]
Jingwen Hou, Henghui Ding, Weisi Lin, Weide Liu, and Yuming Fang. 2022. Distilling Knowledge From Object Classification to Aesthetics Assessment. IEEE Trans. Circuits. Syst. Video Technol. 32, 11 (2022), 7386--7402.
[11]
Jingwen Hou, Weisi Lin, Guanghui Yue, Weide Liu, and Baoquan Zhao. 2022. Interaction-Matrix Based Personalized Image Aesthetics Assessment. IEEE Trans. Multimedia (2022).
[12]
Jingwen Hou, Sheng Yang, and Weisi Lin. 2020. Object-level Attention for Aesthetic Rating Distribution Prediction. In Proc. ACM Int. Conf. Multimedia. 816--824.
[13]
Xin Jin, Hao Lou, Heng Huang, Xinning Li, Xiaodong Li, Shuai Cui, Xiaokun Zhang, and Xiqiao Li. 2022. Pseudo-Labeling and Meta Reweighting Learning for Image Aesthetic Quality Assessment. IEEE Trans. Intell. Transp. Syst. 23, 12 (2022), 25226--25235.
[14]
Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z. Wang, Jia Li, and Jiebo Luo. 2011. Aesthetics and Emotions in Images. IEEE Signal Processing Magazine 28, 5 (2011), 94--115.
[15]
Yueying Kao, Ran He, and Kaiqi Huang. 2017. Deep Aesthetic Quality Assessment With Semantic Information. IEEE Trans. Image Process. 26, 3 (2017), 1482--1495.
[16]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proc. Int. Conf. Learn. Represent. 1--15.
[17]
Shu Kong, Xiaohui Shen, Zhe L. Lin, Radomír Mech, and Charless C. Fowlkes. 2016. Photo Aesthetics Ranking Network with Attributes and Content Adaptation. In Proc. Eur. Conf. Comput. Vis. 662--679.
[18]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proc. Adv. Neural Inf. Process. Syst. 1097--1105.
[19]
Michal Kucer, Alexander C. Loui, and David W. Messinger. 2018. Leveraging Expert Feature Knowledge for Predicting Image Aesthetics. IEEE Trans. Image Process. 27, 10 (2018), 5100--5112.
[20]
Jun-Tae Lee and Chang-Su Kim. 2019. Image Aesthetic Assessment Based on Pairwise Comparison: A Unified Approach to Score Regression, Binary Classification, and Personalization. In Proc. IEEE Int. Conf. Comput. Vis. 1191--1200.
[21]
Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang. 2019. Fast A3RL: Aesthetics-Aware Adversarial Reinforcement Learning for Image Cropping. IEEE Trans. Image Process. 28, 10 (2019), 5105--5120.
[22]
Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, and Weisi Lin. 2020. Personality-assisted Multi-task Learning for Generic and Personalized Image Aesthetics Assessment. IEEE Trans. Image Process. 29 (2020), 3898--3910.
[23]
Yaohui Li, Yuzhe Yang, Huaxiong Li, Haoxing Chen, Liwu Xu, Leida Li, Yaqian Li, and Yandong Guo. 2022. Transductive Aesthetic Preference Propagation for Personalized Image Aesthetics Assessment. In Proc. ACM Int. Conf. Multimedia. 896--904.
[24]
Xin Lu, Zhe Lin, Xiaohui Shen, Radomír Mech, and James Z. Wang. 2015. Deep Multi-patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation. In Proc. IEEE Int. Conf. Comput. Vis. 990--998.
[25]
Xin Lu, Zhe L. Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2015. Rating Image Aesthetics Using Deep Learning. IEEE Trans. Multimedia 17, 11 (2015), 2021--2034.
[26]
Pei Lv, Jianqi Fan, Xixi Nie, Weiming Dong, Xiaoheng Jiang, Bing Zhou, Mingliang Xu, and Changsheng Xu. 2023. User-Guided Personalized Image Aesthetic Assessment Based on Deep Reinforcement Learning. IEEE Trans. Multimedia 25 (2023), 736--749.
[27]
Pei Lv, Meng Wang, Yongbo Xu, Ze Peng, Junyi Sun, Shi-Mei Su, Bing Zhou, and Mingliang Xu. 2018. USAR: An Interactive User-specific Aesthetic Ranking Framework for Images. In Proc. ACM Int. Conf. Multimedia. 1328--1336.
[28]
Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2408--2415.
[29]
Bowen Pan, Shangfei Wang, and Qisheng Jiang. 2019. ImageAestheticAssessment Assisted by Attributes through Adversarial Learning. In Proc. AAAI Int. Conf. Artif. Intell. 679--686.
[30]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Proc. Adv. Neural Inf. Process. Syst. Workshop. 1--4.
[31]
Jian Ren, Xiaohui Shen, Zhe L. Lin, Radomír Mech, and David J. Foran. 2017. Personalized Image Aesthetics. In Proc. IEEE Int. Conf. Comput. Vis. 638--647.
[32]
Dongyu She, Yu-Kun Lai, Gaoxiong Yi, and Kun Xu. 2021. Hierarchical Layout-Aware Graph Convolutional Network for Unified Aesthetics Assessment. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 8475--8484.
[33]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learn. Represent., Yoshua Bengio and Yann LeCun (Eds.). 1--14.
[34]
Wei-Tse Sun, Ting-Hsuan Chao, Yin-Hsi Kuo, and Winston H. Hsu. 2017. Photo Filter Recommendation by Category-Aware Aesthetic Learning. IEEE Trans. Multimedia 19, 8 (2017), 1870--1880.
[35]
Xiaoou Tang, Wei Luo, and Xiaogang Wang. 2013. Content-Based Photo Quality Assessment. IEEE Trans. Multimedia 15, 8 (2013), 1930--1943.
[36]
Guolong Wang, Junchi Yan, and Zheng Qin. 2018. Collaborative and Attentive Learning for Personalized Image Aesthetic Assessment. In Proc. Int. Joint Conf. Artif. Intell. 957--963.
[37]
Weining Wang, Junjie Su, Lemin Li, Xiangmin Xu, and Jiebo Luo. 2019. Meta-Learning Perspective for Personalized Image Aesthetics Assessment. In Proc. IEEE Int. Conf. Image Process. 1875--1879.
[38]
Munan Xu, Jia-Xing Zhong, Yurui Ren, Shan Liu, and Ge Li. 2020. Context-aware Attention Network for Predicting Image Aesthetic Subjectivity. In Proc. ACM Int. Conf. Multimedia. 798--806.
[39]
Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. 2021. Deep Multi-View Enhancement Hashing for Image Retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 43, 4 (2021), 1445--1451.
[40]
Jufeng Yang, Dongyu She, Yu-Kun Lai, Paul L. Rosin, and Ming-Hsuan Yang. 2018. Weakly Supervised Coupled Networks for Visual Sentiment Analysis. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 7584--7592.
[41]
Jiachen Yang, Yanshuang Zhou, Yang Zhao, Wen Lu, and Xinbo Gao. 2022. MetaMP: Metalearning-Based Multipatch Image Aesthetics Assessment. IEEE Trans. Cybern. (2022).
[42]
Shuai Yang, Jiaying Liu,Wenhan Yang, and Zongming Guo. 2019. Context-Aware Text-Based Binary Image Stylization and Synthesis. IEEE Trans. Image Process. 28, 2 (2019), 952--964.
[43]
Yuzhe Yang, Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo. 2022. Personalized Image Aesthetics Assessment With Rich Attributes. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 19861--19869.
[44]
Xiaodan Zhang, Xinbo Gao, Wen Lu, and Lihuo He. 2019. A Gated Peripheral-Foveal Convolutional Neural Network for Unified Image Aesthetic Prediction. IEEE Trans. Multimedia 21, 11 (2019), 2815--2826.
[45]
Yu Zhang, Chang-Bin Zhang, Peng-Tao Jiang, Ming-Ming Cheng, and Feng Mao. 2021. Personalized Image Semantic Segmentation. In Proc. IEEE Int. Conf. Comput. Vis. 10549--10559.
[46]
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning Deep Features for Discriminative Localization. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2921--2929.
[47]
Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. 2022. Personalized Image Aesthetics Assessment via Meta-Learning With Bilevel Gradient Optimization. IEEE Trans. Cybern. 52, 3 (2022), 1798--1811.
[48]
Hancheng Zhu, Yong Zhou, Leida Li, Yaqian Li, and Yandong Guo. 2023. Learning Personalized Image Aesthetics From Subjective and Objective Attributes. IEEE Trans. Multimedia 25 (2023), 179--190.
[49]
Hancheng Zhu, Yong Zhou, Zhiwen Shao, Wenliang Du, Guangcheng Wang, and Qiaoyue Li. 2022. Personalized Image Aesthetics Assessment via Multi-Attribute Interactive Reasoning. Mathematics 10 (2022), 4181.

Cited By

View all
  • (2024)Attribute-Driven Multimodal Hierarchical Prompts for Image Aesthetic Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681175(2399-2408)Online publication date: 28-Oct-2024
  • (2024)"Special Relativity" of Image Aesthetics Assessment: a Preliminary Empirical PerspectiveProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681172(2554-2563)Online publication date: 28-Oct-2024
  • (2024)AesMamba: Universal Image Aesthetic Assessment with State Space ModelsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681011(7444-7453)Online publication date: 28-Oct-2024
  • Show More Cited By

Index Terms

  1. Personalized Image Aesthetics Assessment with Attribute-guided Fine-grained Feature Representation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. aesthetic preferences
    2. attribute-guided fine-grained features
    3. image aesthetics assessment
    4. refined local features

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)145
    • Downloads (Last 6 weeks)10
    Reflects downloads up to 05 Mar 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Attribute-Driven Multimodal Hierarchical Prompts for Image Aesthetic Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681175(2399-2408)Online publication date: 28-Oct-2024
    • (2024)"Special Relativity" of Image Aesthetics Assessment: a Preliminary Empirical PerspectiveProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681172(2554-2563)Online publication date: 28-Oct-2024
    • (2024)AesMamba: Universal Image Aesthetic Assessment with State Space ModelsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681011(7444-7453)Online publication date: 28-Oct-2024
    • (2024)AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics PerceptionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680649(5911-5920)Online publication date: 28-Oct-2024
    • (2024)ReFID: Reciprocal Frequency-aware Generalizable Person Re-identification via Decomposition and FilteringACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364368420:7(1-20)Online publication date: 25-Apr-2024
    • (2024)Hybrid CNN-transformer based meta-learning approach for personalized image aesthetics assessmentJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10404498:COnline publication date: 16-May-2024
    • (2024)Advancing neural aesthetic assessment of artistic images based on bundle features integrationThe Visual Computer10.1007/s00371-024-03732-5Online publication date: 10-Dec-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media