A fast and efficient conformal regressor with regularized extreme learning machine

Abstract

A conformal regressor combines conformal prediction and a traditional regressor for point predictions. It produces a valid prediction interval for a new testing input such that the probability of the target output being not included in the prediction interval is not more than a preset significance level. Although conformal prediction is both theoretically and empirically valid, one main drawback of the existing conformal regressors is their computational inefficiency. This paper proposes a novel fast and efficient conformal regressor named LW-JP-RELM, with combination of the local-weighted jackknife prediction (LW-JP), a new variant of conformal prediction, and the regularized extreme learning machine (RELM). The development of our learning algorithm is important both for the applications of extreme learning machine and conformal prediction. On the one hand, LW-JP-RELM complements ELM with interval predictions that satisfy a given level of confidence. On the other hand, the underlying learning process and the outstanding learning ability of RELM make LW-JP-RELM a very fast and informationally efficient conformal regressor. In the experiments, the empirical validity and informational efficiency of our method were compared to those of the state-of-art on 20 public data sets and the results confirmed that LW-JP-RELM is a competitive and promising conformal regressor.

Introduction

Extreme learning machine(ELM) addresses the question of how to train the feed-forward neural networks fast without losing the learning ability and predicting performance. It is one of the fastest and most popular learning algorithms nowadays and owns the features that the parameters of the hidden nodes of ELM are randomly assigned without being tuned and the output weights are analytically determined [1], [2], [3].

The basic ELM was first proposed by Huang et al. in 2004 [4]. Since then, many variants of ELM have been developed to improve the ideas from many point of view. To make ELM more stable, Huang et al. employed the Tikhonov regularization to ELM which is called regularized ELM [3]. Chen et al. came up with a method to modify the sigmoid activation function [5]. Yuan et al. proposed a new way to solve the output weights [6]. To make ELM more flexible, generalize well and be able to adjust its structure automatically and dynamically, different kinds of incremental ELMs and pruned ELMs are proposed and studied by many researchers. The representative works are CI-ELM [7] and OP-ELM [8]. Recently proposed PCI-ELM, EPCI-ELM and DCI-ELM are more flexible and dynamic methods of this field [9]. To tackle the overfitting problem and take advantage of different ELMs, various kinds of ELM ensembles are studied and examined [10], [11]. To improve the unsupervised learning ability of ELM, a lot of related unsupervised learning methods are also proposed, such as self-organizing maps based extreme learning machine [12], [13]. Moreover, numerous techniques, such as Bayesian analysis, evolutionary computation and fuzzy computation, are combined with ELM to produce many variants of ELM and make further improvement of the original ELM [14], [15], [16], [17], [18], [19], [20], [21].Because of its fast learning speed and excellent performance with little human intervene [22], [23], ELM has attracted more and more attention recently and has progressed dramatically over the past decade, from theoretical analysis [24], [25], [26], [27], [28] to practical applications across many walks of life, such as medical applications, image processing, system modeling and so on [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40]. For the readers who are interested in the history and insights of ELM, we recommend [41] and [42].

To date, almost all of the ELMs for regression have been designed only for point predictions, i.e. they provide a point prediction for every new data [36]. However, in many regression problems, not only point predictions, but also confidence intervals for new data are needed [43], [44], [45], [46]. Moreover, especially in some risk sensitive situations, the probability of the true label being not included in the prediction intervals should be under control, which means the probability should be not more than a significance level preset by human. This property of an interval predictor is called validity [47]. Although a few recent works of ELM have been proposed to tackle this problem, their methods are both based on Bayesian approaches and some prior assumptions of the parameters are needed [43], [44]. If the prior assumptions are correct in a particular application, Bayesian methods can give satisfying confidence intervals and they enjoy the property of being valid, but if not, the intervals output for the new data cannot be trusted, which has been shown in many experiments such as those in [48], [49]. As such, some methods other than the bayesian framework should be employed to make ELM produce valid prediction intervals, which is the first motivation for proposing the learning algorithm in this paper.

In this paper, we only combine the regularized ELM (RELM) instead of other variants of ELMs and conformal prediction to build the conformal regressor. There are three reasons for this choice. First, RELM learns fast, performs well in various kinds of data sets and applications, which is a good option as the underlying algorithm of conformal prediction. Second, the efficient computation of the leave-one-out predictions on the training set of RELM makes the whole learning framework learn even faster. Third, as the first paper to combine the ELM regressor and conformal prediction, RELM is a good choice to make the first attempt and to inspire and motivate more future works.

Conformal prediction [49], [50] is a learning method designed to complement the predictions of traditional learning algorithms, which are called underlying algorithms, with confidence measures. A combination of conformal prediction with a particular underlying algorithm for regression is a conformal regressor, which can produce prediction intervals for the inputs of the testing data. Under the standard assumption of the data being sampled independently and identically distributed, any conformal regressor is valid [49]. Since the framework of conformal prediction was developed, many conformal regressors have been proposed and shown to be empirically valid and useful in many real world applications [45], [46], [51], [52], [53]. Although conformal regressors are strongly guaranteed to be valid theoretically and empirically, the main drawback of the existing conformal regressors is their computational inefficiency, which results from the computational framework of the conformal prediction and the learning time of the underlying learning algorithms [45], [46].

To overcome the computational inefficiency exhibited by the conformal prediction, one way is to modify the learning framework of it. Up to now, some modifications of the framework have been proposed , including inductive conformal prediction [45], cross-conformal prediction [46], split conformal prediction and jackknife prediction [54]. Inductive conformal prediction and split conformal prediction also have solid theoretical bases and are more computationally efficient than the original conformal prediction, but they lose some of the informational efficiency as they partition the training set into two parts for the two stages of conformal prediction process. Cross-conformal prediction and jackknife prediction make full use of training data. Although not theoretically valid, they are shown to be empirically valid and informationally efficient in the empirical studies. Another way to speed up the learning process is to employ a fast and accurate learning method as the underlying algorithm, which is the reason of employing extreme learning machine in this paper.

Besides the computational efficiency, another issue of implementing the conformal prediction in the real world is the informational efficiency, which is reflected by the average interval length produced by the conformal regressor and whether the length varies along with the variance of the target output. The shorter the average interval length is and the more consistent the interval length and the variance of the target output are, the more information a conformal regressor sends. To make the prediction interval more informational, local-weighted conformal inference, which can also be called local-weighted conformal prediction, has been proposed based on the local-weighted residuals [54]. Local-weighted conformal inference needs another learning algorithm to estimate the variance of the target output given the input vector, which is another computational burden of regression conformal prediction.

Although the extreme learning machine learns very quickly, to our best knowledge, there are no studies using the extreme learning machine as the underlying algorithm to accelerate the computation of conformal prediction.

In order to get the best of both worlds, this paper proposes a novel conformal regressor named LW-JP-RELM which is a combination of locally-weighted jackknife prediction (LW-JP) [54] and RELM. Although the original approach and other variants of conformal prediction can also be combined with RELM, there is actually an additional advantage from using LW-JP. To be specific, the intervals produced by LW-JP are calculated based on the leave-one-out predictions of the underlying algorithm on the training set. Unlike many other underlying algorithms for which it takes long to obtain the leave-one-out predictions on the training set, RELM can calculate the predictions very fast [55]. In effect, the computational complexity of obtaining the predictions is equal to that of training RELM, which means that the predictions are just bonus for RELM when the training data are large and LW-JP-RELM can be very fast.

All in all, the leaning characters of RELM and the computational framework of LW-JP make the two methods the best combination and we propose it originally and examine it empirically in this paper.

The rest of this paper is organized as follows. Section 2 first briefly introduces conformal prediction, and then presents the learning framework of local-weighted jackknife prediction, the framework used in this paper. Section 3 reviews the regularized extreme learning machine and the formulas of the leave-one-out prediction and error of it. In Section 4, LW-JP-ELM is developed and the computational complexity is presented. Experiments are introduced in Section 5 and eventually, in Section 6, the conclusions of this paper are drawn.