Pairwise similarity plays a dominate role for image classification. However, the semantic consistency along image pairs is often ignored. This hinders the classification accuracy, especially when we have few supervision information. In this article, we go one step beyond pairwise similarity and propose a novel SCG-MV similarity method for image classification. We use the divided intervals between two images for semantic coherence measurement. Multiview information is also used to improve the classification accuracy. The proposed method can be combined with other similarity based methods. Extensive experiments prove the effectiveness of the proposed method. To the best of our knowledge, this is the first work which explores semantic coherence between images for classification instead of only using pairwise similarity.

Pairwise similarity has been widely used for image classification by propagating the class information from labeled images to unlabeled images and predicting the classes of unlabeled images accordingly. Although widely used, pairwise similarity based classification methods have two drawbacks. On one hand, visual-semantic similarity consistency does not always hold due to semantic gap. On the other hand, the reliability of pairwise similarity is also influenced by the number and underlying distribution of labeled images. A well-designed classification model cannot be easily generalized to few-shot classification tasks. To solve these two problems, in this article, we propose a semantic coherence guided multiview similarity (SCG-MV) for image classification method with varied supervision levels. We measure the semantic coherence between image pairs for classifier learning. This is achieved by first conducting line search along the line segment between each image pair, and then measure th...