Ordinal extreme learning machine

Abstract

Recently, a new fast learning algorithm called Extreme Learning Machine (ELM) has been developed for Single-Hidden Layer Feedforward Networks (SLFNs) in G.-B. Huang, Q.-Y. Zhu and C.-K. Siew “[Extreme learning machine: theory and applications,” Neurocomputing 70 (2006) 489–501]. And, ELM has been successfully applied to many classification and regression problems. In this paper, the ELM algorithm is further studied for ordinal regression problems (named ORELM). We firstly proposed an encoding-based framework for ordinal regression which includes three encoding schemes: single multi-output classifier, multiple binary-classifications with one-against-all (OAA) decomposition method and one-against-one (OAO) method. Then, the SLFN was redesigned for ordinal regression problems based on the proposed framework and the algorithms are trained by the extreme learning machine in which input weights are assigned randomly and output weights can be decided analytically. Lastly widely experiments on three kinds of datasets were carried to test the proposed algorithm. The comparative results with such traditional methods as Gaussian Process for Ordinal Regression (ORGP) and Support Vector for Ordinal Regression (ORSVM) show that ORELM can obtain extremely rapid training speed and good generalization ability. Especially when the data set’s scalability increases, the advantage of ORELM will become more apparent. Additionally, ORELM has the following advantages, including the capabilities of learning in both online and batch modes and handling non-linear data.

Introduction

In machine learning, the classification and metric regression are two important supervised learning problems. Classification is a learning task that assigns labels to instances that belong to a finite set of object classes, while metric regression predicts one value in one continuous distinction for new instances. Ordinal regression, a setting bridging between classification and metric regression, is a learning task of predicting variables of ordinal scale. In contrast to metric regression problems, ordinal regression often requires the grades be discrete and finite. These grades are also different from the class labels in classification problems due to the existence of ranking information. The good ordinal regression methods should not only obtain good accuracy, but also hold the fast training speed and responding speed, the ability to deal with linear and non-linear problems and good online learning ability.

Many machine learning algorithms have been proposed or redesigned for ordinal regression [1], [2], including the perceptron [3] and its kernelized generalization [4], the neural network with gradient descent [2], [6], the Gaussian process [7], [8], the large margin classifier [10], [11], [29], the k-partite classifier [12], the boosting algorithm [13], [14], the constraint classification [15], regression trees [16], Naive Bayes [17], Bayesian hierarchical experts [18], the binary classification approach [19], [20], and the optimization of nonsmooth cost functions [21]. Although these ordinal regression methods have some interesting properties, they have also certain disadvantages. For example, Prank [14] is a fast online algorithm whose accuracy suffers when dealing with non-linear data. Large-margin methods [22] are accurate approaches, but they convert the ordinal relations into O(N²) (N is the number of data points) pairwise ranking constraints, and thus, are impractical for large size datasets. Ordinal support vector machines (ORSVM) [11], [10] are another powerful large-margin methods finding m−1 real value thresholds (m is the number of ranks), their optimization problem's complexity is O(N), and the prediction speed is slow when the support vector is not sparse. Similarly, the Gaussian process method for ordinal regression (ORGP) [7] has also the difficulty of handling large size datasets. Therefore, some better methods are expected to solve the ordinal regression problem.

Most of these ordinal regression algorithms are based on iterative learning for thresholds, for example, ORGP is the combination of Gaussian process (GP) with threshold-based learning, and ORSVM is the combination of support vector machine (SVM) with threshold-based learning. These threshold-based approaches are always relatively complex and long consuming for iterative learning steps. If we want to enhance learning speed, we have to give up this iterative learning approach. The encoding approach has been proved to be effective for multi-classification problems by experimental results reported in Allwein’s paper [9]. Allwein only discussed the encoding framework for multi-classification problems (called extended correcting output coding, ECOC), ordinal regression problem is different with classification problem as the ordinal relationship of categories exists. Therefore, while designing the encodings for classification, it need not consider this ordinal relationship. But, for ordinal regression, it is necessary to consider this point. In this paper, we proposed three encoding schemes for ordinal regression that can reflect these ordinal relationships. They are different with the coding for multi-classification. For example, considering of the order of the categories, if a data point x belongs to the kth rank, it is automatically classified into lower-order categories (1, 2, …, k−1). So the target output of x is encoded as t=[1, 1, ..., 1, −1, −1, ..., −1] where t_i(1≤i≤k) is set to “1” and other elements “−1”. While in classification problem, if a data point x belongs to the kth category, it is will not be able to be classified into another category. So the target output of x will be encoded as t=[−1, −1, ..., 1, −1, −1, ..., −1]. This is our first contribution. This proposed framework can be combined with SVM, RVM and so on. In order to get a fast learning algorithm, we select the Extreme Learning Machine [23] and propose three ELM-based ordinal regression methods through designing three different output encoding schemes [9] and the corresponding model structure: learning with a single multi-output ELM (called ORELM-SingleELM), multi-binary ELM with One-against-All scheme (called ORELM-OAA) and multi-binary ELM with One-against-One scheme (called ORELM-OAO). This is our second attribution. Lastly, we analyze and test the proposed schemes by comparing with popular thresholds-based ones, and show that our scheme is very fast and is especially suitable for real-time application, e.g., user mode track and personalized recommendation. These proposed methods origins ELM's advantages, such as learning in both online [28], [30], [31], [32], [33] and batch mode [23], training on very large datasets, handling non-linear data, and extremely rapid training speed. Some comparative experiments will be done to show its performances.

The rest of the paper is arranged as follows. In Section 2, some related work, including ELM and ECOC, are introduced. The proposed ELM schemes for ordinal regression are presented in detail in Section 3, which include the ORELM-SingleELM, ORELM-OAA and ORELM-OAO schemes. In Section 4, the schemes' performances are evaluated and compared with existing ones. Finally, some conclusions are drawn and discussions are given in Section 5.