A semi-supervised online sequential extreme learning machine method

Abstract

This paper proposes a learning algorithm called Semi-supervised Online Sequential ELM, denoted as SOS-ELM. It aims to provide a solution for streaming data applications by learning from just the newly arrived observations, called a chunk. In addition, SOS-ELM can utilize both labeled and unlabeled training data by combining the advantages of two existing algorithms: Online Sequential ELM (OS-ELM) and Semi-Supervised ELM (SS-ELM). The rationale behind our algorithm exploits an optimal condition to alleviate empirical risk and structure risk used by SS-ELM, in combination with block calculation of matrices similar to OS-ELM. Efficient implementation of the SOS-ELM algorithm is made viable by an additional assumption that there is negligible structural relationship between chunks from different times. Experiments have been performed on standard benchmark problems for regression, balanced binary classification, unbalanced binary classification and multi-class classification by comparing the performance of the proposed SOS-ELM with OS-ELM and SS-ELM. The experimental results show that the SOS-ELM outperforms OS-ELM in generalization performance with similar training speed, and in addition outperforms SS-ELM with much lower supervision overheads.

Introduction

In recent years, Extreme Learning Machine (ELM), proposed by Huang et al. [1], [2], [3], [4], is attracting more and more attention because of its outstanding performance in training speed, predicting accuracy and generalization ability [5], [6], [7], [8], [9], [10]. In particular, it is shown that ELM tends to outperform support vector machine (SVM) in both regression and classification applications with much easier implementation [11]. However, batch ELM is still a time consuming affair, although much faster than traditional learning algorithms including SVM. Time cost is mainly reflected in two aspects: (1) Matrix inversion calculation, whose computational complexity is between quadratic and cubic with respect to the training data size [12]; (2) Output weight updating, which consumes considerable time as batch ELM needs to do recalculation by combining the old data together with the new data when a new chunk of training data is received. Online Sequential ELM (OS-ELM) [13] adapted batch ELM to overcome the abovementioned problems for practical applications. OS-ELM learns the training data chunk-by-chunk and updates the output weight only by new training data. Thus OS-ELM not only saves storage but also decreases the computational complexity. In addition, unlike other sequential learning algorithms which have many control parameters to be tuned, OS-ELM only requires the number of hidden nodes to be specified. In a word, OS-ELM presents much more advantage than many traditional sequential learning algorithms, like SGBP [14], RAN [15], RANEKF [16], MRAN [17], [18], GAP-RBF [19] and GGAP-RBF [20].

Although OS-ELM has advantages in generalization performance and learning speed, it still cannot avoid the dependency on a large amount of labeled data, which usually involves high cost in labor and time. Compared with OS-ELM, semi-supervised learning methods benefit from utilizing unlabeled data and reducing the need for labeled data. Compared with labeled data, unlabeled data is much easier and cheaper to acquire. Therefore semi-supervised learning provides an effective solution for the problems with small amount of labeled samples in various classification and regression tasks. To exploit unlabeled data, some semi-supervised ELM variants [21], [22], [23], [24] have been proposed. As typical examples, [22], [24] propose a kind of semi-supervised ELM based on manifold regularization, so that the learning system can balance the empirical risk and the complexity of the learned function f, where [22] is an improvement of [24] in terms of semi-supervised ELM, which brings good performance on predicting accuracy. However, the semi-supervised ELM mentioned above learns in a batch way, so its training speed decreases rapidly as the sample size gets larger.

To solve the manual labeling cost problem and meet the demand of sequential learning for many real applications, we propose a new type of online sequential ELM, which we name as semi-supervised online sequential ELM (SOS-ELM). It inherits not only the training and testing speed of OS-ELM but also the prediction accurate of SS-ELM. The experimental results show that using the same number of labeled samples, our proposed SOS-ELM has higher accurate prediction rate than that of OS-ELM and much faster training speed than that of SS-ELM.

The details of our proposed SOS-ELM are elaborated in the remainder part of the paper which is organized as follows. Section 2 gives a brief review of the batch ELM, OS-ELM and SS-ELM. Section 3 illustrates the derivation of SOS-ELM. Section 4 presents the experimental results and discussion based on the benchmark problems in the areas of regression and classification. Conclusions and future work based on the study are given in Section 5.