Error analysis of distributed least squares ranking

Abstract

Learning theory of distributed kernel methods has attracted much attentions recently. However, the existing theory analysis is limited to the kernel regression with pointwise losses. It is not clear whether theory guarantees can be obtained for distributed kernel methods with pairwise losses. To answer this question, this paper considers a new pairwise ranking algorithm, called distributed regularized least squares ranking (DRLSRank), under the divide and conquer strategy. Rather than minimizing the empirical pairwise risk associated the whole data, the proposed DRLSRank obtains the individual ranking functions based on data subsets and takes their weighted average as a final predictor. Theoretically, we derive the generalization bounds in expectation via integral operator approximation technique. Our results show that DRLSRank can achieve a satisfactory learning rate and the additional unlabeled data are crucial for relaxing the restriction on the number of data subsets, which fills the theoretical gap on learning theory for distributed pairwise ranking.

Introduction

Divide and conquer strategy has been used successfully for kernel methods with big scale data [1], [2], [3]. Compared with other techniques for big scale data, e.g., Markov sampling [4], [5], [6] and path algorithm [7], [8], divide and conquer strategy explores all the data in small batches simultaneously. In general, there are three key steps for kernel methods under divide and conquer strategy: dividing the whole data into manageable subsets, obtaining the individual learning machine in each subset, and getting the final predictor by combining these individual machines [2], [3], [9]. Following this way, distributed learning has provided an effective strategy to conquer the big data challenges [1], [2], [10] and to realize the privacy-preserving data mining [3].

Along with its practical success, theoretical foundations of distributed kernel methods have been investigated recently. For distributed regularized least-squares, the optimal learning rates in expectation are provided in [2], [9]. For the distributed spectral algorithm, the convergence rate has been well understood in [11]. In [12], the error analysis for distributed multi-penalty regularization is derived by Neumann expansion and a second order decomposition on the difference of operator inverse approach. In [13], the generalization ability is characterized for a distributed gradient descent algorithm. A novel analysis framework is conducted for distributed semi-supervised regression [3], which shows the unlabeled data play important roles in reducing the distributed error and relaxing the restriction on the number of data subsets. In [14], the bias correction is applied to further improve the learning performance of regularized kernel networks, and their optimal learning rates are reached for distributed regression setting.

Despite rapid progress on distributed learning theory, all the above results are restricted to the pointwise kernel methods (or kernel methods with pointwise losses). It still remains unclear for learning theory analysis to distributed kernel methods with pairwise losses (e.g., pairwise ranking [15], [16], [17], [18], similarity/metric learning [19], [20], [21]). Meanwhile, the computational complexity of pairwise learning machines is usually higher than the corresponding pointwise learning, especially for the big data setting. This motivates us to explore theoretic foundations for pairwise kernel methods under divide and conquer strategy.

By applying the divide and conquer strategy to regularized least squares ranking (RLSRank) [15], [16], [22], we formulate a distributed ranking algorithm, called distributed regularized least squares ranking (DRLSRank). Under this distributed learning strategy, the proposed ranking model has much advantage on computational feasibility and can deal with ranking task with big data. The main contribution of this paper is to establish the generalization bounds of DRLSRank based on the solution characteristics of RLSRank [16], [17], [23], the operator approximation techniques [24], and the error decomposition strategy in [3]. The error bounds show the proposed DRLSRank can also achieve satisfactory learning rates under mild conditions, which provide the learning theory guarantees for the distributed pairwise approach.

The remainder of this paper includes the algorithm framework and its theoretical analysis. In Section 2, we briefly review some related works on regularized least squares ranking and ranking learning theory. Section 3 formulates the distributed regularized ranking and Section 4 provides the upper bound on the excess ranking risk. Finally, Section 5 concludes the paper.