Deep matrix factorization with knowledge transfer for lifelong clustering and semi-supervised clustering

Abstract

Clustering analysis aims to group unlabeled data in an unsupervised learning manner. However, most existing methods are tailored for single-task data and do not work for a sequence of tasks. In this paper, we propose a Deep Matrix factorization method with Knowledge transfer (DMK) to address clustering problem in a lifelong setting, where DMK approaches a sequence of tasks; after each task is learned, its knowledge will be retained and later used to help future clustering task. To this end, we delve into deep matrix factorization and graph co-clustering, where (1) the former learns a basis feature library across all arrived tasks and a specific representation for each target task to deal with lifelong clustering and (2) the latter builds a consistent feature embedding library to transfer knowledge between each pair of tasks. An iterative optimization algorithm is then proposed to alternatively update the two libraries. In addition, we extend our DMK into a semi-supervised version and propose a Semi-supervised Deep Matrix factorization method with Knowledge transfer (SDMK) by exploiting a few of prior label information for lifelong semi-supervised clustering. Experimental results using four datasets with sequential tasks demonstrate that the proposed methods outperform state-of-the-art baseline methods markedly.

Introduction

Clustering analysis is one of the techniques that partition a set of data objects in such a way that data objects in the same group (or called cluster) are more similar to each other than to those in other groups. In recent years, data clustering has been studied extensively from some novel perspectives, say clustering with noisy data [13], incomplete data clustering [17], multi-view clustering [36], and multi-task clustering [48], [49]. Among these, multi-task clustering transfers the knowledge learned from different tasks to help clustering for a target task and groups the data of multiple tasks at the same time. For example, Zhang et al. [47] proposed a multi-task clustering method to transfer instance knowledge across tasks to help cluster each target task. Zhang et al. [48] also proposed a model to jointly mine task relationship among all tasks to help data clustering of every task. However, multi-task clustering methods mainly consider a fixed task set and do not learn incrementally. Given a sequence of tasks, when a new task is fed into a multi-task clustering model, the model needs to rebuild on all arrived tasks repeatedly, which is costly and time-consuming as the tasks are possibly endless. In this paper, we focus on the problem of data clustering of a sequence of tasks. This problem setting is similar as the current lifelong learning setting [5], which addresses a set of clustering tasks consecutively. Lifelong learning is a continual learning process where the learner learns a sequence of tasks; after learning each task, the learner retains knowledge and later uses the knowledge to help learn future task [2]. Mostly existing works focus on lifelong classification problems. In this work, we concern a lifelong clustering problem.

Rao et al. [25] proposed an unsupervised continual learning model to learn a task specific representation on the top of a larger set of shared parameters, which aims to solve catastrophic forgetting in neural networks. Note that the data in [25] are actually from the same task as the same dataset is partitioned into multiple subsets and then each subset is treated as a task. While, our tasks are from different datasets (see experiment section). Sun et al. [29] proposed a lifelong spectral clustering method to investigate spectral clustering in a lifelong learning setting. However, this method only extracts the shallow information of the data using spectral graph theory and cannot mine the deep intrinsic information among data samples. Our lifelong clustering method is a deep model, which is based on deep matrix factorization to effectively extract deep information [18], [3]. Meanwhile, these methods work in an unsupervised manner without any label information. In practice, a few of prior label information can robust and boost clustering performance remarkably [11], [10], [16]. Learning with partially supervised label information is known as semi-supervised learning [34]. To our knowledge, there is no work on lifelong semi-supervised clustering. In our work, we bring semi-supervised learning into our lifelong clustering framework and further propose a lifelong semi-supervised clustering method. In brief, the framework of our lifelong clustering is shown in Fig. 1.

Regarding clustering in a lifelong learning setting, there are at least two key challenges need to address. (1) How to mine shared knowledge among multiple tasks? Similar to multi-task learning, it needs to extract effective and sufficient information across tasks and to learn the correlation between tasks. (2) How to learn and transfer knowledge from previous tasks to the target/new task? The number of tasks in lifelong learning is possibly never ending and the task set is not fixed unlike multi-task learning. Thus, it would be essential that storing knowledge learned from previous tasks and transferring the knowledge to a new task to boost clustering performance for this new task.

In this paper, we design a deep multi-layer architecture to learn deep-layer representations across a sequence of tasks and meanwhile utilize graph co-clustering to transfer knowledge from one task to another. We then propose a Deep Matrix factorization method with Knowledge transfer (denoted as DMK) for lifelong clustering based on deep matrix factorization and graph co-clustering. Given a set of prior label information in each task, we further propose a Semi-supervised DMK (denoted as SDMK) for lifelong semi-supervised clustering. The proposed DMK and SDMK will be clear shortly. In summary, this paper makes the following contributions:

• It studies a new clustering paradigm called lifelong clustering, which aims to retain and accumulate knowledge learned from each previous task and to use the knowledge to help future clustering tasks.

• It proposes a deep multi-layer model (i.e., Deep Matrix factorization method with Knowledge transfer) for lifelong clustering. The proposed model learns a basis feature library shared by all previous tasks and meanwhile learns a task-specific representation for each new/target task. It also exploits graph co-clustering to learn a common feature embedding library to exchange and transfer latent information between sequential tasks.

• It further proposes a lifelong semi-supervised clustering method by exploiting a few of prior label information, which uses a label knowledge library to transfer semi-supervised knowledge among tasks.

• An alternating iterative optimization algorithm is presented to optimize the proposed lifelong clustering and semi-supervised clustering methods respectively in the lifelong setting. Experimental results on real-world datasets demonstrate the effectiveness and efficiency of the proposed methods.