Cluster-based adaptive SVM: A latent subdomains discovery method for domain adaptation problems

Highlights

• We propose a new latent subdomains discovery method for domain adaptation approach.

• We call this new method CA-SVM and it is designed based on a new simultaneously clustering and adaptation method.

• CA-SVM uses linear SVM model for classification in the source and target domains.

• CA-SVM does not need source samples for applying division.

Abstract

Machine learning algorithms often suffer from good generalization in testing domains especially when the training (source) and test (target) domains do not have similar distributions. To address this problem, several domain adaptation techniques have been proposed to improve the performance of the learning algorithms when they face accuracy degradation caused by the domain shift problem. In this paper, we focus on the non-homogeneous distributed target domains and propose a new latent subdomain discovery model to divide the target domain into subdomains while adapting them. It is expected that applying adaptation on subdomains increase the rate of detection in comparing with the situation that the target domain is seen as one single domain. The proposed division method considers each subdomain as a cluster which has the definite ratio of positive to negative samples, linear discriminability and conditional distribution similarity to the source domain. This method divides the target domain into subdomains while adapting the trained target classifier for each subdomain using Adapt-SVM adaptation method. It also has a simple solution for selecting the appropriate number of subdomains. We call our proposed method Cluster-based Adaptive SVM or CA-SVM in short. We test CA-SVM on two different computer vision problems, pedestrian detection and image classification. The experimental results show the advantage in accuracy rate for our approach in comparison to several baselines.

Introduction

In many real-world applications, it is common that the distribution of training data is different from that of test data. This distribution shift is mainly caused by factors such as difference in illumination, viewpoint, background, resolution, etc. The distribution shift between two domains is inevitable and caused by the process of collecting data. Many studies have demonstrated the influence of shift between distribution of training and test domains in accuracy degradation (Farhadi and Tabrizi, 2008, Saenko et al., 2010, Torralba and Efros, 2011).

Domain Adaptation (DA) is a branch of machine learning, proposed to address domain shift problems (Gopalan et al., 2011, Hoffman et al., 2013, Mirrashed and Rastegari, 2013, Pan et al., 2008, Pan et al., 2011). When the training (source) and test (target) domains have different but related distributions, DA performs adaptation between source and target domains such that it increases the detection accuracy in the target domain. In order to come along with adaptation challenge, a variety of methods have been proposed based on different assumptions about the source and target domains. One branch of adaptation studies focus on the problems that have access to the enough number of labeled source samples and a few number of labeled target samples (Hoffman et al., 2013, Aytar and Zisserman, 2011, Jiang et al., 2008, Xu et al., 2014b, Xu et al., 2014a, Duan et al., 2012b). In this case, adaptation is performed by finding the common space in which the source and target samples have similar distributions and can be used for training an appropriate classifier (Saenko et al., 2010, Donahue et al., 2013, Gong et al., 2012, Gopalan et al., 2011, Hoffman et al., 2013, Mirrashed and Rastegari, 2013, Pan et al., 2008, Pan et al., 2011, Yeh et al., 2014). The common space is found by the information obtained from the source and target domains based on available samples. This approach is more accurate than other approaches and can define adaptation more precisely. However, the time and memory complexity of this approach is high which makes it not applicable for all kinds of problems (especially for detection problems with huge number of source samples). Along with this group, there are some specific adaptation methods which use the trained source model and a few available target samples for adaptation (Aytar and Zisserman, 2011, Jiang et al., 2008, Mozafari and Jamzad, 2014, Xu et al., 2014b, Xu et al., 2014a, Yang et al., 2007b, Yang et al., 2007a). The methods in this group do not have access to the source samples. Instead they try to use the trained source model as a means of adaptation. The variation of methods in this branch is limited, and most of them apply adaptation in the original space of the source and target domains. The reason for applying adaptation in the original space comes back to the lack of information that a source model has to describe the source domain. This information is essential for finding the common space. The variations of model-transferring adaptation methods are limited. However, they are very important because of their low time and memory complexity (Mozafari and Jamzad, 2016) and they are well suited for real-time applications.

Most of DA methods consider the source and target domains as two homogeneous distributed domains and apply adaptation between them. Sometimes, source, target or both are made of distributionally non-homogeneous domains. A good example for a non-homogeneous domain is a constructed domain by the collected images from a search engine. Because a search engine returns data from different resources, therefore the searched images have different distributions. Fig. 1 shows the 13 top images collected by searching “cat” word in Google search engine. The returned images can be divided into three subdomains according to their difference in viewpoint, background and illumination.

To perform adaptation for distributionally non-homogeneous domains, the first step is dividing the source and target domains into distributionally homogeneous subdomains and then applying adaptation between each related subdomains. Accordingly, subdomain division is the main concern in these cases. Several methods (Duan et al., 2012b, Hoffman et al., 2012) are proposed for dividing the source and target domains into distributionally related subdomains. The common characteristics between them are their dependency on available source and target samples for dividing the source and target domains into appropriate subdomains for adaptation.

The focus of this paper, is on proposing a new adaptation method for non-homogeneous target domains which need to be adapted using a linear SVM trained source model. We select model-transferring approach for adaptation and propose a new clustering method for subdomain division that we call it Cluster-based Adaptive SVM or CA-SVM for short. CA-SVM uses a new proposed method to divide the target domain into homogeneous target subdomains. The proposed clustering method tries to keep the balance of labeled samples, linearly discriminable characteristic of samples and distribution similarity of subdomains to the source domain, in each cluster. CA-SVM divides the target domain and simultaneously applies adaptation for each subdomain using Adapt-SVM method (Yang et al., 2007a). We apply CA-SVM on two test-beds of pedestrian detection and image classification. We will highlight the role of separating the subdomains in improving the adaptation success of our proposed method in comparison to several baselines that see the target domain as a one-complete domain and to the baselines that divide the target domain into subdomains.

The rest of paper is organized as follows. In Section 2, we review the related works which mainly is focused on SVM-based adaptation methods. Section 3 contains a short introduction to SVM-based model-transferring methods followed by the details of the proposed CA-SVM algorithm and its analysis. The experimental results on real and synthetic data are shown in Section 4 and finally we draw the conclusion and future work in Section 5.