Abstract
Existing deep learning-based image emotion analysis methods regard image emotion classification as a usual classification task in which the semantics of categories are clear. Nevertheless, the semantics of emotion categories are fuzzy, leading to that people are ambiguous between emotions of similar semantic distance when observing images. Considering the semantic distance of emotion categories, that is, far or near distance relations between them, we design a similarity decline rule to first pre-process the similarities of sample pairs making them comparable. Then, image emotion analysis is performed through deep metric learning. For key issues in deep metric learning, that is, sampling and weighting, we design adaptive decision boundaries for sampling and a double-weighted mechanism for sampled pairs which is integrated in our proposed emotion constraint loss, which learns more information contributing to update model by boasting the weights. Therefore, more expressive embedding features are learned from embedding space. Thus, the similarity of pairs from adjacent categories is larger than that from far away ones. The experimental results demonstrate that our proposed method outperforms the state-of-the-art methods. In addition, the ablation experiments show that it is necessary to consider the semantic distance of emotion categories in image emotion analysis.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Detenber, B.H., Simons, R.F., Bennett, G.G., Jr.: Roll’em!: the effects of picture motion on emotional responses. Broadcast. Electr. Media 42(1), 113–127 (1998)
Mikels, J., Fredrickson, A., Larkin, B.L., et al.: Emotional category data on images from the international affective picture system. Behav. Res. Methods 37(4), 626–630 (2005)
Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: International Conference on Multimedia, pp. 83–92. ACM, Firenze (2010)
Zhao, S., Gao, Y., Jiang, X., et al.: Exploring principles-of-art features for image emotion recognition. In: The ACM International Conference on Multimedia, pp. 47–56. ACM, Orlando (2014)
Borth, D., Ji, T., Chen, T., et al.: Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: ACM Multimedia Conference, pp. 223–232. ACM, Barcelona (2013)
Zhao, S., Zhao, X., Ding, G., et al.: Emotiongan: unsupervised domain adaptation for learning discrete probability distributions of image emotions. In: ACM Multimedia Conference on Multimedia Conference, pp. 1319–1327. ACM, Seoul (2018)
Deng, J., Dong, W., Socher, R., et al.: ImageNet: a large-scale hierarchical image database. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, Florida (2009)
Peng, K.C., Chen, T., Sadovnik, A., et al.: A mixed bag of emotions: model, predict, and transfer emotion distributions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 860–868. IEEE, Boston (2015)
Yang, J., Sun, M., Sun, X.: Learning visual sentiment distributions via augmented conditional probability neural network. In: AAAI Conference on Artificial Intelligence, pp. 224–230. AAAI, California (2017)
Zhao, S., Ding, G., Gao, Y., et al.: Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans. Affect. Comput. 11(4), 574–587 (2020)
Geng, X.: Label distribution learning. IEEE Trans. Knowl. Data Eng. 28(7), 1734–1748 (2016)
Yang J., She D., Sun M.: Joint image emotion classification and distribution learning via deep convolutional neural network. In: International Joint Conference on Artificial Intelligence, pp. 3266–3272. Melbourne (2017)
Xiong, H., Liu, H., Zhong, B., et al.: Structured and sparse annotations for image emotion distribution learning. In: AAAI Conference on Artificial Intelligence, pp. 363–370. AAAI, Hawaii (2019)
Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1735–1742. IEEE, New York (2006)
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Conference on Learning Representations, pp. 84–92. Sprinter, San Diego (2015)
Oh Song, H., Xiang, Y., Jegelka, S., et al.: Deep metric learning via lifted structured feature embedding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4004–4012. IEEE, Las Vegas (2016)
Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. In: Advances in Neural Information Processing Systems, pp. 1849–1857. Barcelona (2016)
Yi, D., Lei, Z., Li, S.Z.: Deep metric learning for practical person re-identification. arXiv:1407.4979 (2014)
Wang, X, Han, X.T., Huang, W.L., et al.: Multi-similarity loss with general pair weighting for deep metric learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5022–5030. IEEE, Long Beach (2019)
Sun, Y., Cheng, C., Zhang, Y., et al.: Circle loss: a unified perspective of pair similarity optimization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6397–6406. IEEE, Seattle (2020)
Yang, J., She, D, Lai, Y., et al.: Retrieving and classifying affective images via deep metric learning. In: AAAI Conference on Artificial Intelligence, pp. 491–498. AAAI, New Orleans (2018)
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant No. 61163019 and No. 61540062, the Yunnan Applied Basic Research Key Project under Grant No. 2014FA021, and the Scientific Research Project of Yunnan Province Education Department under Grant No. 2021J0029 and 2021Y027.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, G., Zhang, H., Xu, D. (2021). Image Emotion Analysis Based on the Distance Relation of Emotion Categories via Deep Metric Learning. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-89029-2_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89028-5
Online ISBN: 978-3-030-89029-2
eBook Packages: Computer ScienceComputer Science (R0)