Skip to main content
SpringerLink
Log in

Affective image classification via semi-supervised learning from web images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Affective image classification has drawn increasing research attentions in the affective computing and multimedia communities. Despite many solutions proposed in the literature, it remains a major challenge to bridge the semantic gap between visual features of images and their affective characteristics, partly due to the lack of adequate training samples, which can be largely ascribed to the all-consuming nature of affective image annotation. In this paper, we propose a novel affective image classification algorithm based on semi-supervised learning from web images (SSL-WI). This algorithm consists of four major steps, including color and texture feature extraction, baseline classifier construction, feature selection, and jointly using training images and retrieved web images to re-train the classifier. We have applied this algorithm, the baseline classifier that is not trained by web images, and two state-of-the-art algorithms to differentiating color images in a three-dimensional discrete emotional space. Our results suggest that, with the scheme of semi-supervised learning from web images, the proposed algorithm is able to produce more accurate affective image classification than other three approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Baidu CBIR system-Baidu Institute of Deep Learning. http://idl.baidu.com/IDL-news-2.html

  2. Bleschke M, Madonski R, Rudnicki R (2009) Image retrieval system based on combined MPEG-7 texture and colour descriptors. In: 2009 16th international conference on mixed design of integrated circuits & systems, pp 635–639

  3. Bosse T, Zwanenburg E (2014) Do prospect-based emotions enhancebelievability of game characters? A casestudy in the context of a dice game. IEEE Trans Affect Comput 5(1):17–31

    Article  Google Scholar 

  4. Bovik AC, Clark M, Geisler WS (1990) Multichannel texture analysis using localized spatial filters. IEEE Trans Pattern Anal Mach Intell 12(1):55–73

    Article  Google Scholar 

  5. Chen H, Gao Z, Lu G, Li S (2008) A novel support vector machine fuzzy network for image classification using MPEG-7 visual descriptors. In: 2008 international conference on multimedia and information technology (MMIT), pp 365–368

  6. Chen YY, Chen T, Liu T, Liao HYM, Chang SF (2015) Assistive image comment robot—a novel mid-level concept-based representation. IEEE Trans Affect Comput 6(3):298–311

    Article  Google Scholar 

  7. Cieplinski L, Kim M, Ohm JR, Pickering M, Yamada A (2001) Text of ISO/IEC 15938-3/FCD information technology multimedia content description interface part 3 visual: ISO/IEC JTC1/SC29/ WG11/N4062

  8. Colombo C, Del Bimbo A, Pala P (1999) Semantics in visual information retrieval. IEEE MultiMed 6(3):38–53

    Article  Google Scholar 

  9. Cozman FG, Cohen I, Cirelo MC (2003) Semi-supervised learning of mixture models. In: Proceedings of the 20th international conference on machine learning, pp 99–106

  10. Dai D, Van Gool L (2013) Ensemble projection for semi-supervised image classification. In: 2013 IEEE international conference on computer vision (ICCV), pp 2072–2079

  11. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 886–893

  12. Dorairaj R, Namuduri KR (2004) Compact combination of MPEG-7 color and texture descriptors for image retrieval. In: 2004 conference record of the thirty-eighth Asilomar conference on signals, systems and computers, vol 1, pp 387–391

  13. Duan R, Jiang W, Man H (2006) Semi-supervised image classification in likelihood space. In: 2006 IEEE international conference on image processing (ICIP), pp 957–960

  14. Geusebroek JM (2006) Compact object descriptors from local colour invariant histograms. Brit Mach Vis Conf 3:1029–1038

    Google Scholar 

  15. Houle ME, Oria V, Satoh SI, Sun J (2013) Annotation propagation in image databases using similarity graphs. ACM Trans Multimed Comput Commun Appl 10(1):1–21

    Article  Google Scholar 

  16. Huang K (2008) Colorful natural scenes retrieval based on affective features hierarchical model. In: 2008 international symposium on information science and engineering (ISISE), vol 2, pp 183–187

  17. Itten J (1992) The art of color (Chinese version). Shanghai People’s Art Press, Shanghai

    Google Scholar 

  18. Itten J (1992) Design and form-the basic course at the Bauhous (Chinese version). Shanghai People’s Art Press, Shanghai

    Google Scholar 

  19. Joonwhoan L, EunJong P (2011) Fuzzy similarity-based emotional classification of color images. IEEE Trans Multimed 13(5):1031–1039

    Article  Google Scholar 

  20. Joshi D, Datta R, Fedorovskaya E, Luong QT (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28(5):94–115

    Article  Google Scholar 

  21. Kawamto N, Soen T (1993) Objective evaluation of color design II. Color Res Appl 18:260–266

    Article  Google Scholar 

  22. Le THN, Luu K, Savvides M (2013) SparCLeS: dynamic sparse classifiers with level sets for robust beard/moustache detection and segmentation. IEEE Trans Image Process 22

  23. Lee PM, Hsiao TC (2014) Applying LCS to affective image classification in spatial-frequency domain. In: 2014 IEEE congress on evolutionary computation (CEC), pp 1690–1697

  24. Li HF, Jin Q (2008) Research of image affective semantic rules based on neural network. In: 2008 international seminar on future biomedical information engineering (FBIE), pp 148–151

  25. Li YF, Zhou ZH (2015) Towards making unlabeled data never hurt. IEEE Trans Pattern Anal Mach Intell 37(1):175–188

    Article  Google Scholar 

  26. Li HF, Li J, Song JC (2007) Extracting emotional semantics from color image using analytical hierarchy process. In: 2007 third international conference on intelligent information hiding and multimedia signal processing (IIHMSP), vol 1, pp 612–618

  27. Li HF, Wen X, Jin H (2008) The clustering algorithm research of image emotional characteristics based on ant colony. In: 2008 international symposium on intelligent information technology application workshops (IITAW), pp 455–458

  28. Li N, Xia Y, Xia Y (2015) Semi-supervised emotional classification of color images by learning from cloud. In: 2015 international conference on affective computing and intelligent interaction (ACII), pp 84–90

  29. Liu Z, Yu X (2012) Identification of image emotional semantic based on feature fusion. In: 2012 international conference on computer science & service system (CSSS), pp 1802–1806

  30. Liu W, Jiang Y-G, Luo J, Chang S-F (2011) Noise resistant graph ranking for improved web image search. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 849–856

  31. Machajdik J, Hanbury A (2010) Affective Imag classification using features inspired by psychology and art theory. In: Proceedings of the international conference on multimedia. ACM, pp 83–92

  32. Merdassi H, Barhoumi W, Zagrouba E (2014) Semi-supervised image classification in large datasets by using random forest and fuzzy quantification of the salient object. In: 2014 international workshop on computational intelligence for multimedia understanding (IWCIM), pp 1–5

  33. Ou LC, Luo MR, Woodcock A, Wright A (2010) A study of colour emotion and colour preference. Part I: colour emotions for single colours. Color Res Appl 29 (3):232–240

    Article  Google Scholar 

  34. Park TS, Zhang BT (2015) Consensus analysis and modeling of visual aesthetic perception. IEEE Trans Affect Comput 6(3):272–285

    Article  Google Scholar 

  35. Smith JR, Chang SF (1996) A fully automated content-based image query system. ACM Multimed

  36. Soen T, Shimada T, Akita M (1987) Objective evaluation of color design. Color Res Appl 12:187–195

    Article  Google Scholar 

  37. Solli M, Lenz R (2011) Color emotions for multi-colored images. Color Res Appl 36(3):210–221

    Article  Google Scholar 

  38. Song X, Peng X, Xu J, Wu F (2014) Cloud-based distributed image coding. In: 2014 IEEE international conference on image processing (ICIP), pp 4802–4806

  39. Sung-Bae C (2004) Emotional image and musical information retrieval with interactive genetic algorithm. Proc IEEE 92(4):702–711

    Article  Google Scholar 

  40. Tang J, Hong R, Yan S, Chua T-S, Qi G-J, Jain R (2011) Image annotation by k nn-sparse graph-based label propagation over noisily tagged web images. ACM Trans Intell Syst Technol (TIST) 2(2):14

    Google Scholar 

  41. Tang JH, Zha ZJ, Tao D, Chua TS (2012) Semantic-gap-oriented active learning for multilabel image annotation. IEEE Trans Image Process 21(4):2354–2360

    Article  MathSciNet  Google Scholar 

  42. Tian GJ, Fu H, Feng DD (2008) Automatic medical image categorization and annotation using LBP and MPEG-7 edge histograms. 2008 international conference on information technology and applications in biomedicine (ITAB), pp 51–53

  43. Tkalcic M, Odic A, Kosir A, Tasic J (2013) Affective labeling in a content-based recommender system for images. IEEE Trans Multimed 15(2):391–400

    Article  Google Scholar 

  44. Wang WN, He QH (2008) A survey on emotional semantic image retrieval. In: 2008 15th IEEE international conference on image processing (ICIP), pp 117–120

  45. Wang J, Liu B (2010) A study on emotion classification of image based on bp neural network. In: 2010 international conference of information science and management engineering (ISME), vol 1, pp 100–104

  46. Wang S, Wang X (2005) Emotion semantics image retrieval: an brief overview. In: 2005 international conference on affective computing and intelligent interaction (ACII), pp 490–497

  47. Wang WN, Yu YL (2005) Image emotional semantic query based on color semantic description. In: Proceedings of 2005 international conference on machine learning and cybernetics, vol 7, pp 4571–4576

  48. Wang WN, Yu YL, Zhang JC (2004) Image emotional classification: static vs. dynamic. In: 2004 IEEE international conference on systems, man and cybernetics, vol 7, pp 6407–6411

  49. Wang WN, Yu YL, Jiang SM (2006) Image retrieval by emotional semantics: a study of emotional space and feature extraction. In: 2006 IEEE international conference on systems, man and cybernetics (SMC), vol 4, pp 3534–3539

  50. Wang X, Jia J, Yin J (2013) Interpretable aesthetic features for affective image classification. In: 2013 20th IEEE international conference on image processing (ICIP), pp 3230–3234

  51. Wang Q, Meng Z, Li X (2017) Locality adaptive discriminant analysis for spectral–spatial classification of hyperspectral images. IEEE Geosci Remote Sens Lett (GRSL) 14(11):2077–2081

    Article  Google Scholar 

  52. Wang Q, Wan J, Yuan Y (2017) Deep metric learning for crowdedness regression. IEEE Trans Circuits Syst Video Technol (T-CSVT) 99:1–1

    Google Scholar 

  53. Wang Q, Wan J, Yuan Y (2018) Locality constraint distance metric learning for traffic congestion detection. Pattern Recogn 75:272–281

    Article  Google Scholar 

  54. Xie WX, Lu ZW, Peng YX, Xiao JG (2013) Multimodal semi-supervised image classification by combining tag refinement, graph-based learning and support vector regression. In: 2013 20th IEEE international conference on image processing (ICIP), pp 4307–4311

  55. Yamada A, Pickering M, Jeannin S, Cieplinski L, Ohm JR, Kim M (2001) MPEG-7 visual part of experimentation model version 10.0: ISO/IEC JTC1/SC29/WG11/N4063

  56. Yannakakis GN, Isbister K, Paiva A, Karpouzis K (2014) Guest editorial: emotion in games. IEEE Trans Affect Comput 5(1):1–2

    Article  Google Scholar 

  57. Yanulevskaya V, van Gemert JC, Roth K (2008) Emotional valence categorization using holistic image features. In: 2008 15th IEEE international conference on image processing (ICIP), pp 101–104

  58. Zhang X, Ren F (2008) Improving Svm learning accuracy with Adaboost. In: 2008 fourth international conference on natural computation (ICNC), vol 3, pp 221–225

  59. Zhao Y, Wang B, Ye S, Zheng Y, Shi L (2009) Image retrieval based on improved MCD and EHD of MPEG-7. In: 2009 international conference on test and measurement (ICTM), vol 1, pp 127–132

  60. Zhao J, Lu H, Li Y, Chen J (2012) A kind of fuzzy decision tree based on the image emotion classification. In: 2012 international conference on computing, measurement, control and sensor network (CMCSN), pp 167–170

  61. Zhao S, Gao Y, Jiang X, Yao H, Chua T-S, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the 22nd ACM international conference on multimedia, pp 47–56

  62. Zhao M, Zhan C, Wu Z, Tang P (2015) Semi-supervised image classification based on local and global regression. IEEE Signal Process Lett 22(10):1666–1670

    Article  Google Scholar 

  63. Zhao S, Han Y, Zou Q, Hu Q (2016) Hierarchical support vector machine based structural classification with fused hierarchies. Neurocomputing 214(Supplement C):86–92

    Article  Google Scholar 

  64. Zhu S, Sun X, Jin D (2016) Multi-view semi-supervised learning for image classification. Neurocomputing 208(Supplement C):136–142

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61771397 and 61471297, and in part by the Seed Foundation of Innovation and Creation for Graduate Students in Northwestern Polytechnical University under Grants Z2016150.

Author information

Authors and Affiliations

  1. Shaanxi Key Lab of Speech & Image Information Processing (SAIIP), School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an, 710072, China

    Na Li & Yong Xia

  2. Centre for Multidisciplinary Convergence Computing (CMCC), School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an, 710072, China

    Yong Xia

Authors
  1. Na Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Xia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, N., Xia, Y. Affective image classification via semi-supervised learning from web images. Multimed Tools Appl 77, 30633–30650 (2018). https://doi.org/10.1007/s11042-018-6131-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6131-1

Keywords

Search

Navigation