Cross-modal social image clustering and tag cleansing

Highlights

• Semantic image clustering for exploit image topics of interest.

• Kernel combination for cross-modal contexts among weekly-tagged images.

• K-way min-max cut for social image clustering.

Abstract

In this paper, a cross-modal approach is developed for social image clustering and tag cleansing. First, a semantic image clustering algorithm is developed for assigning large-scale weakly-tagged social images into a large number of image topics of interest. Spam tags are detected automatically via sentiment analysis and multiple synonymous tags are merged as one super-topic according to their inter-topic semantic similarity contexts. Second, multiple base kernels are seamlessly combined by maximizing the correlations between the visual similarity contexts and the semantic similarity context, which can achieve more precise characterization of cross-modal (semantic and visual) similarity contexts among weakly-tagged social images. Finally, a K-way min–max cut algorithm is developed for social image clustering by minimizing the cumulative inter-cluster cross-modal similarity contexts while maximizing the cumulative intra-cluster cross-modal similarity contexts. The optimal weights for base kernel combination are simultaneously determined by minimizing the cumulative within-cluster variances. The polysemous tags and their ambiguous images are further split into multiple sub-topics for reducing their within-topic visual diversity. Our experiments on large-scale weakly-tagged Flickr images have provided very positive results.

Introduction

Collaborative image tagging systems, such as Flickr.com, have now become very popular for tagging large-scale social images by relying on collaborative efforts of a large population of Internet users [1], [2]. In a collaborative image tagging system, people may tag social images according to their social or cultural backgrounds, personal expertise and perception. We call such collaboratively-tagged social images as weakly-tagged social images because their social tags may not have exact correspondences with the underlying image semantics. With the exponential growth of weakly-tagged social images, it has become increasingly attractive to develop new algorithms for achieving more effective organization and summarization of large-scale social images.

Image clustering, which can assign large amounts of images into different clusters with some common semantics or visual properties [2], [33], [34], [35], [41], [42], [43], [44], [45], is very attractive for achieving more effective organization, summarization and visualization of large-scale image collections. Clustering the search results into multiple semantic groups is helpful for users to assess the relevance between large amounts of returned images (which are returned by the same query terms) and their query intentions [33], [34], [35], [44]. Image clustering has also been used to achieve more precise organization and summarization of large-scale Flickr images according to their visual similarities [2]. Most existing algorithms for image clustering focused on only the low-level visual features [44], thus it is doubtful of their effectiveness because of the problem of semantic gap [55], [56], [57], [58]. Some researchers have recently integrated visual and textual features for Web image clustering and higher accuracy rates are reported [33], [34], [35], [41], [42], [43].

In a collaborative image tagging space, there are two inter-related information sources that can be used to support image clustering: (1) visual properties of weakly-tagged social images; and (2) their social tags. It is worth noting that the visual properties of weakly-tagged social images and their social tags can offer complementary strengths, thus it is very attractive to develop new frameworks that are able to integrate the visual features with the social tags for achieving more precise clustering of large-scale weakly-tagged social images. Unfortunately, it is not a trivial work because of the following issues:

(a) Social Tag Ambiguity: Different people may use different social tags (i.e., text terms), which have the same or close meaning (synonyms), to tag their social images [5], [26], [27], [28], [29], [30], [31]. The weakly-tagged social images, which belong to a set of synonymous tags, may share some common visual properties and semantics. The appearance of synonymous tags may prevent most existing clustering algorithms from deriving more representative image clusters. On the other hand, collaborative image tagging is an ambiguous process [22], [23], [24], [25]. Different people may apply the same social tag in different ways (i.e., the polysemous tag may have different meanings under different contexts), which may result in large amounts of ambiguous images with diverse visual properties. Because the effectiveness of most existing clustering algorithms largely depends on the accuracy of the underlying functions for data similarity characterization, the appearance of polysemous tags and their ambiguous images may bring new challenges for social image clustering, e.g., it is hard to design suitable similarity functions for characterizing the diverse image similarity contexts accurately. In a collaborative image tagging space, dishonest users may use spam tags to tag their social images, so that they can derive traffic to their social images for fun or profit [37], [38], [39]. The appearance of spam tags may result in large amounts of junk images, which may mislead most existing clustering algorithms to derive less representative clusters from large-scale weakly-tagged social images.

(b) Visual Ambiguity and Semantic Gap: Multiple types of visual features are usually extracted to achieve more sufficient characterization of various visual properties of the images, thus the distributions of the images could be very sparse and the visual similarity contexts among the images could be very diverse in the high-dimensional feature space (i.e., visual ambiguity). As a result, it is very hard to use one single type of base kernels such as RBF kernel [7], [8], [9], [10], [11], [12] to achieve precise characterization of the diverse visual similarity contexts among the images. In addition, there may have large amounts of outliers in the high-dimensional feature space and most existing clustering algorithms may seriously suffer from the problem of skewed cuts. Another challenging problem for image clustering is the semantic gap [55], [56], [57], [58] between the low-level visual features and the image semantics, e.g., it is very hard to achieve semantic clustering of large-scale weakly-tagged social images by using only the low-level visual features. It is worth noting that the visual properties of weakly-tagged social images and their social tags can offer complementary strengths, thus they can be integrated to achieve more precise clustering of large-scale weakly-tagged social images. Because the visual features and the social tags belong to different spaces, it is unsatisfied to combine them directly for social image clustering.

As shown in Fig. 1, a cross-modal approach is developed in this paper to achieve social image clustering and tag cleansing: (a) a semantic image clustering algorithm is developed for extracting the image topics of interest from large amounts of social tags; (b) spam tags are ide.epsied automatically via sentiment analysis and multiple synonymous tags are merged as one super-topic according to their inter-topic semantic similarity contexts; (c) a mixture-of-kernels algorithm is developed to achieve more accurate characterization of the cross-modal similarity contexts among the weakly-tagged social images; (d) a K-way min–max cut algorithm is extended for supporting cross-modal social image clustering and tag cleansing, where the polysemous tags and their ambiguous images are split into multiple sub-topics for reducing their intra-topic visual diversity; (e) a topic network is constructed to achieve more effective organization and summarization of large-scale weakly-tagged social images at the semantic level.

The rest of this paper is organized as follows. In Section 2, a brief review of some relevant work is presented; In Section 3, a semantic image clustering algorithm is introduced to assign large-scale social images into a large number of image topics of interest; In Section 4, a mixture-of-kernels algorithm is developed for achieving more precise characterization of the diverse cross-modal image similarity contexts among the social images; In Section 5 K-way min–max cut algorithm is presented for achieving cross-modal social image clustering; In Section 6 topic network is constructed to enable semantic summarization and organization of large-scale weakly-tagged social images at the semantic level; Our experimental results on algorithm evaluation are given in Section 7 and we conclude this paper in Section 8.