Abstract
Convolutional neural networks (CNNs) simulate the structure and function of the nervous system based on biological characteristics. CNNs have been used to understand the emotions that images convey. Most existing studies of emotion analysis have focused only on image emotion classification, and few studies have paid attention to relevant regions evoking emotions. In this paper, we solve the issues of image emotion classification and emotional region localization based on weakly supervised deep learning in a unified framework. We train a fully convolutional network, followed by our proposed cross-spatial pooling strategy, to generate an emotional activation map (EAM), which represents the relevant region that could evoke emotion in an image and is only labelled with an image-level annotation. Extensive experiments demonstrate that our proposed method has the best performance in the accuracy of classification and emotional region localization.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Peng, K.C., Chen, K.C., Sadovnik, A., Gallagher, A.: A mixed bag of emotions: model, predict, and transfer emotion distributions. In: The 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, pp. 860–868. IEEE Press (2015)
You, Q.Z., Luo, J.B., Jin, H.L., Yang, J.C.: Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: The 30th Conference on Artificial Intelligence, Arizona, pp. 308–314. IEEE Press (2016)
You, Q.Z., Luo, J.B., Jin, H.L., Yang, J.C.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: The 29th Conference on Artificial Intelligence, Austin, pp. 381–388. IEEE Press (2015)
Victor, C., Brendan, J., Giró-i-Nieto, X.: From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction. Image Vis. Comput. 65, 15–22 (2017)
Peng, K.C., Sadovnik, A., Gallagher, A., Chen, T.: Where do emotions come from? Predicting the emotion stimuli map. In: The 2016 IEEE International Conference on Image Processing, Phoenix, pp. 614–618. IEEE Press (2016)
Yang, J.F., She, D.Y., Lai, Y.k., Yang, M.H.: Retrieving and classifying affective images via deep metric learning. In: the 30th innovative Applications of Artificial Intelligence, New Orleans, pp. 491–498. IEEE Press (2018)
Borth, D., Ji, R.R., Chen, T., et al.: Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: The 2013 ACM Multimedia Conference, Barcelona, pp. 223–232. ACM Press (2013)
Chen, T., Borth, D., Darrell, T, et al.: DeepSentibank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586 (2014)
Ali, A.R., Shahid, U., Ali, M., et al.: High-level concepts for affective understanding of images. In: The 2017 IEEE Winter Conference on Applications of Computer Vision, Santa Rosa, pp. 678–687. IEEE Press (2017)
Kosti, R., Alvarez, J.M., Recasens A., et al.: Emotion recognition in context. In: The 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, pp. 1960–1968. IEEE Press (2017)
Bilen, H., Vedaldi, A.: Weakly supervised deep detection networks. In: The 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, pp. 2846–2854. IEEE Press (2016)
Cinbis, R.G., Verbeek, J., Schmid, C.: Weakly supervised object localization with multi-fold multiple instance learning. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 189–203 (2016)
Zhou, B.L, Aditya, K., Agata, L., Aude, O., Antonio, T.: Learning deep features for discriminative localization. In: The 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, pp. 2921–2929. IEEE Press (2016)
Selvaraju, R.R, Cogswell, M., Das, A., et al.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: The 2017 IEEE International Conference on Computer Vision, Venice, pp. 618–626. IEEE Press (2017)
Durand, T., Mordan, T., Thome, N., et al.: WILDCAT: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. In: The 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, pp. 5957–5966. IEEE Press (2017)
Zhu, Y., Zhou, Y., Ye Q., et al.: Soft proposal networks for weakly supervised object localization. In: The 2017 IEEE International Conference on Computer Vision, Italy, pp. 1859–1868. IEEE Press (2017)
Yang, J.F., She. D.Y., Lai, Y.K., Rosin, P.L.: Weakly supervised coupled networks for visual sentiment analysis. In: The 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, pp. 7584–7592. IEEE Press (2018)
Fan, S.J., Shen, Z.Q., Jiang, M., Koening L., et al.: Emotional attention: a study of image sentiment and visual attention. In: The 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, pp. 7521–7531. IEEE Press (2018)
Paszke, A., Gross, S., Chintala, S., et al.: Pytorch. https://pytorch.org/ (2017)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant Nos. 61163019 and No. 61540062, the Yunnan Applied Basic Research Key Project under Grant No. 2014FA021, and the Yunnan Provincial Education Department’s Scientific Research Fund Industrialization Project under Grant No. 2016CYH03.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, G., Xu, D. (2019). Weakly Supervised Learning of Image Emotion Analysis Based on Cross-spatial Pooling. In: Sun, Z., He, R., Feng, J., Shan, S., Guo, Z. (eds) Biometric Recognition. CCBR 2019. Lecture Notes in Computer Science(), vol 11818. Springer, Cham. https://doi.org/10.1007/978-3-030-31456-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-31456-9_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31455-2
Online ISBN: 978-3-030-31456-9
eBook Packages: Computer ScienceComputer Science (R0)