An active three-way clustering method via low-rank matrices for multi-view data

Highlights

• A method for clustering on multi-view data with high dimensionality.

• Three-way clustering approach to cope with the uncertain relationship between objects and clusters.

• An active learning strategy by taking the advantage of three-way representation of clustering.

• Information fusion by using the low-rank matrix representation and the weighted multiple views.

• The framework is flexible for semi-supervised clustering and unsupervised clustering.

Abstract

In recent years, multi-view clustering algorithms have shown promising performance by combining multiple sources or views of datasets. A problem that has not been addressed satisfactorily is the uncertain relationship between an object and a cluster. Thus, this paper investigates an active three-way clustering method via low-rank matrices that can improve clustering accuracy as clustering proceeds for the multi-view data of high dimensionality. We adopt a three-way clustering representation to reflect the three types of relationships between an object and a cluster, namely, belong-to definitely, uncertain and not belong-to definitely. We construct the consensus low-rank matrix from each weighted low-rank matrix by taking account of the diversity of views, and give the method to solve the optimization problem of objective function based on the improved augmented Lagrangian multiplier algorithm. We suggest an active learning strategy to learn important informative pairwise constraints after measuring the uncertainty of an object based on the entropy concept. The experimental results conducted on real-world datasets have validated the effectiveness of the proposed method.

Introduction

Multi-view data are very common in some scientific data analytics problems such as computer video, social computing and environmental sciences, due to the use of different measuring methods (e.g. infrared and visual cameras), or of different media, like text, video and audio. Multi-view clustering (MVC), which makes use of the complementary information embedded in multiple views to improve clustering performance, has attracted more and more attentions [30]. In the existing methods, spectral clustering is a popular one for multi-view data because it represents multi-view data via graph structure and makes it possible to handle complex data such as high-dimensional and heterogeneous as well as it can easily use the pairwise constraint information provided by users [6], [10], [20], [28].

In some practical applications, particularly in the multimedia domain, a view is usually represented in a high-dimensional feature space. For high-dimensional data, the feature distribution is usually more sparse, the traditional similarity measurement methods based on distance measures become inapplicable. In order to solve this problem, the approaches based on low-rank representation try to learn a common low-dimensional subspace from the high-dimensional multi-view data and each object can be represented linearly by others objects in the same subspace, which contributes to reduce the computation cost and improves the robustness to noise corruptions [27], [32], [44].

On the other hand, we find that a problem of the existing cluster analysis methods has not been addressed satisfactorily is the uncertain relationship between an object and a cluster. It is obviously that there are three relationships between an object and a cluster, namely, belong-to definitely, not belong-to definitely and uncertain. In most of the existing work, a cluster is represented by a single set, the set naturally divides the space into two regions. Objects belong to the cluster if they are in the set, otherwise they do not. Here, only two relationships are considered, no matter in hard clustering or in soft clustering. They are typically based on two-way (i.e., binary) decisions.

Let us obverse the third relationship, which means the object may or may not belong to the cluster. We just cannot make decisions based on the present obtained knowledge or information. We can make further certain decisions when we have further information. It is a typical idea of three-way decisions. Inspired by the theory of three-way decisions as suggested by Yao [35], [36], Yu [37] has introduced a framework of three-way cluster analysis (TWC). The previous results on three-way clustering [38], [40], [41] provide us with a tool for studying the problem of clustering with uncertainty.

In this paper, we focus on a general framework based on the theory of three-way decisions, which is appropriate for soft clustering or hard clustering. This three-way representation with two sets brings more insight into interpretation of clusters. Objects in the core region certainly belong to the cluster, objects in the trivial region definitively do not belong to the cluster, and objects in the fringe region maybe or may not belong to the cluster. Obviously, the two-way representation with a single set is a special case of three-way representation with two sets when fringe regions are empty.

Compared with the supervised learning, clustering process lacks the user guidance or the class label information and may not produce the desired clusters. Thus, some semi-supervised clustering methods are proposed. These methods that use certain weak supervision form, such as pairwise constraints, can significantly improve the quality of unsupervised clustering. Pairwise constraints describe two objects whether they should be assigned to the same cluster or the different clusters. However, choosing the supervised information is random in most of existing methods, and it does not produce positive effect on improving the clustering result when the algorithm itself can find the prior information or there are amounts of noises in the prior information. Therefore, the active learning method is introduced to optimize the selection of the constraints for semi-supervised clustering [26], [29].

Hence, our work consider active learning of constraints in an iterative framework based on spectral clustering. In each iteration, we determine objects with the most important information toward improving the current clustering result and form queries accordingly instead of choosing the information randomly. The responses to the queries (i.e., constraints) are then used to update the clustering. This process repeats until we reach a stable solution or we reach the maximum number of queries allowed. Such an iterative framework is widely used in active learning for semi-supervised clustering. The measurement of information is designed based on the entropy concept. Besides, to take the advantage of three-way representation of clustering, it is reasonable to choose pairwise constraints in fringe regions instead of the universe, which will improve the search efficiency.

In this paper, we address the problem of clustering on multi-view data of high dimensionality. The main contributions are summarized as follows:

• A novel active three-way clustering method via low-rank matrices for multi-view data is proposed. It considers the diversity of multiple views and to improve the quality of clustering for multi-view data. A multi-view information fusion algorithm is presented via low-rank matrix representation for high-dimensional multi-view data, and the weights are adjusted adaptively on each view during solving the optimization problem of objective function.

• A three-way clustering representation is utilized to reflect the three relationships between an object and a cluster, namely, belong-to definitely, uncertain and not belong-to definitely. The three-way clustering approach provides us with a tool for studying the problem of clustering with uncertainty.

• An active three-way clustering algorithm is developed by taking the advantage of three-way representation of clustering, which can produce the three-way results as well as two-way results accordingly. The idea of farthest-first traversal scheme is used to construct the cores of clusters, then to expand cores to fringes by using the idea of k nearest neighbors, and the rules to adjust the consensus similarity matrix are also introduced.

• The proposed method is flexible for semi-supervised clustering as well as unsupervised clustering.

• We evaluate the proposed method with some other algorithms on seven real-world datasets. The results of comparative experiments demonstrate the effectiveness of the proposed method and show that it is appropriate for multi-view data of high dimensionality.

The rest of the paper is organized as follows. In Section 2, we review existing clustering algorithms for multi-view data, clustering approaches for uncertain relationships and active learning approaches for clustering. In Section 3, we give a detailed description of the proposed method termed Active Three-way Clustering via Low-rank Matrices (ATCLM). In Section 4, we report on an extensive experimental evaluation of the proposed method. Finally, we summarize the present study in Section 5.