DLRankSVM: an efficient distributed algorithm for linear RankSVM

Abstract

Linear RankSVM is one of the widely used methods for learning to rank. The existing methods, such as Trust-Region Newton (TRON) method along with Order-Statistic Tree (OST), can be applied to train the linear RankSVM effectively. However, extremely lengthy training time is unacceptable when using any existing method to handle the large-scale linear RankSVM. To solve this problem, we thus focus on designing an efficient distributed method (named DLRankSVM) to train the huge-scale linear RankSVM on distributed systems. First, to efficiently reduce the communication overheads, we divide the training problem into subproblems in terms of different queries. Second, we propose an efficient heuristic algorithm to address the load balancing issue (which is a NP-complete problem). Third, using OST, we propose an efficient parallel algorithm (named PAV) to compute auxiliary variables at each computational node of the distributed system. Finally, based on PAV and the proposed heuristic algorithm, we develop DLRankSVM under the framework of TRON. The extensive empirical evaluations show that DLRankSVM not only can obtain impressive speedups on both multi-core and distributed systems, but also can perform well in prediction compared with the other state-of-the-art methods. To the best of our knowledge, this is the first research work proposed to train the huge-scale linear RankSVM in a distributed fashion.

Appendices

Appendix A: Proof of Theorem 1

Proof

As described in Algorithm 1, any $l^{(s)}$ associated with the first $z-1$ computational nodes should satisfy $|dr(l^{(s)})|\le { error}$. This means that for the first $z-1$ computational nodes, all corresponding $l^{(s)}$ should meet the condition of $l^{(s)}\le (1+{ error})\frac{l}{z}$. Furthermore, the number of tuples $(y_i,q_i,\mathbf {x}_i)$ allocated to the zth computational node should be greater than zero. Therefore, we can get the following inequality:

$$\begin{aligned} l-\sum \limits _{s=1}^{z-1}l^{(s)}\ge l-(z-1)(1+{ error})\frac{l}{z}>0. \end{aligned}$$

(37)

According to above inequality, we have:

$$\begin{aligned} 0\le { error}<\frac{1}{z-1} \end{aligned}$$

(38)

which proves Theorem 1. $\square$

Appendix B: Proof of Theorem 2

Proof

Without lose of generality, we assume that $X_s=[\ldots , X^{(Q_t)},\ldots ]^T$ represents the set of the training instances allocated to the sth computational node. Then, according to (19) and (27), the computational formulas with respect to $A^{(s)T}_wA^{(s)}_wX_s\mathbf {v} , A^{(s)T}_wA^{(s)}_wX_s\mathbf {w} , A^{(s)T}_w\mathbf {e}^{(s)}_w$, and $p^{(s)}_w$ can be, respectively, denoted by:

$$\begin{aligned} A^{(s)T}_wA^{(s)}_wX_s\mathbf {v}= & {} \begin{bmatrix} \vdots \\ \left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {x}^T_{ti}\mathbf {v}-\left( \beta _{ti}^+(\mathbf {w},\mathbf {v})+\beta _{ti}^-(\mathbf {w},\mathbf {v})\right) \\ \vdots \\ \end{bmatrix} \end{aligned}$$

(39)

$$\begin{aligned} A^{(s)T}_wA^{(s)}_wX_s\mathbf {w}= & {} \begin{bmatrix} \vdots \\ \left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{ti}-\left( \gamma _{ti}^+(\mathbf {w})+\gamma _{ti}^-(\mathbf {w})\right) \\ \vdots \\ \end{bmatrix} \end{aligned}$$

(40)

$$\begin{aligned} A^{(s)T}_w\mathbf {e}^{(s)}_w= & {} \begin{bmatrix} \vdots \\ \alpha _{ti}^+(\mathbf {w})-\alpha _{ti}^-(\mathbf {w})\\ \vdots \\ \end{bmatrix} \end{aligned}$$

(41)

$$\begin{aligned} p^{(s)}_w= & {} \sum \limits _{t}{p^t_w}=\sum \limits _{t}\sum \limits _{i=1}^{l_t}\alpha _{ti}^+(\mathbf {w})=\sum \limits _{t}\sum \limits _{i=1}^{l_t}\alpha _{ti}^-(\mathbf {w}). \end{aligned}$$

(42)

Notice that, on the one hand, each computational nodes possesses the same global $\mathbf {w}$ and $\mathbf {v}$ at each same outer or CG iteration of TRON. On the other hand, the discussion in Sect. 3.3 shows that if $\mathbf {x}_i$ is $\mathbf {x}_{tj}$ (or $y_i$ is $y_{tj}$), then $\alpha ^+_i(\mathbf {w}) , \alpha ^-_i(\mathbf {w}) , \beta ^+_i(\mathbf {w},\mathbf {v}) , \beta ^-_i(\mathbf {w},\mathbf {v}) , \gamma ^+_i(\mathbf {w})$, and $\gamma ^-_i(\mathbf {w})$ must be equal to $\alpha ^+_{tj}(\mathbf {w})$, $\alpha ^-_{tj}(\mathbf {w}), \beta ^+_{tj}(\mathbf {w},\mathbf {v}),\beta ^-_{tj}(\mathbf {w},\mathbf {v}), \gamma ^+_{tj}(\mathbf {w})$, and $\gamma ^-_{tj}(\mathbf {w})$, respectively. As a result, using (40)–(42), $\sum _{s=1}^{z}f^{(s)}(\mathbf {w})$ can be indicated as follows:

$$\begin{aligned} \sum \limits _{s=1}^{z}f^{(s)}(\mathbf {w})= & {} \frac{z}{2}\mathbf {w}^T\mathbf {w}+\sum \limits _{s=1}^{z}C\left( \mathbf {w}X_s\left( A^{(s)T}_wA^{(s)}_wX_s\mathbf {w}-2A^{(s)T}_w\mathbf {e}^{(s)}_w\right) +p^{(s)}_w\right) \nonumber \\= & {} \frac{z}{2}\mathbf {w}^T\mathbf {w}+C\sum \limits _{t=1}^{m}\left( \sum \limits _{i=1}^{l_t}\mathbf {w}^T\mathbf {x}_{ti}\left( \left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{ti}\right) \right) \nonumber \\&-C\sum \limits _{t=1}^{m}\left( \sum \limits _{i=1}^{l_t}\mathbf {w}^T\mathbf {x}_{ti}\left( \left( \gamma _{ti}^+(\mathbf {w})+\gamma _{ti}^-(\mathbf {w})\right) \right) \right) \nonumber \\&-2C\sum \limits _{t=1}^{m}\left( \sum \limits _{i=1}^{l_t}\mathbf {w}^T\mathbf {x}_{ti}\left( \alpha _{ti}^+(\mathbf {w})-\alpha _{ti}^-(\mathbf {w}\right) \right) +C\sum \limits _{t=1}^{m}\left( p^t_w\right) \nonumber \\= & {} \frac{z}{2}\mathbf {w}^T\mathbf {w}+C\sum \limits _{i=1}^{l}\mathbf {w}^T\mathbf {x}_{i}\left( \left( \alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{i}\right) \nonumber \\&-C\sum \limits _{i=1}^{l}\mathbf {w}^T\mathbf {x}_{i}\left( \gamma _{i}^+(\mathbf {w})+\gamma _{i}^-(\mathbf {w})\right) \nonumber \\&-2C\sum \limits _{i=1}^{l}\mathbf {w}^T\mathbf {x}_{i}\left( \alpha _{i}^+(\mathbf {w})-\alpha _{i}^-(\mathbf {w})\right) +Cp_w\nonumber \\= & {} \frac{z-1}{2}\mathbf {w}^T\mathbf {w}+f(\mathbf {w}). \end{aligned}$$

(43)

Consequently,

$$\begin{aligned} f(\mathbf {w})=\sum \limits _{s=1}^{z}f^{(s)}(\mathbf {w})-\frac{z-1}{2}\mathbf {w}^T\mathbf {w}. \end{aligned}$$

To facilitate the proof of the other two equations, we assume that $\mathbf {x}_i$ and $\mathbf {x}_{ti}$ are, respectively, indicated as below:

$$\begin{aligned} \mathbf {x}_{ti}= & {} \left[ x^1_{ti},\ldots ,x^n_{ti}\right] \\ \mathbf {x}_i= & {} \left[ x^1_i,\ldots ,x^n_i\right] . \end{aligned}$$

Then, by plugging (40) and (41) into $\sum _{s=1}^{z}\nabla f^{(s)}(\mathbf {w})$, we have

$$\begin{aligned}&\sum \limits _{s=1}^{z}\nabla f^{(s)}(\mathbf {w})=z\mathbf {w}+2C\sum \limits _{s=1}^{z}X^T_s\left( A^{(s)T}_wA^{(s)}_wX_s\mathbf {w}-A^{(s)T}_w\mathbf {e}^{(s)}_w\right) \nonumber \\&\quad =z\mathbf {w}+2C\nonumber \\&\quad \begin{bmatrix} \sum \limits _{t=1}^{m}\sum \limits _{i=1}^{l_t}x^1_{ti}\left( \left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{ti}-\left( \gamma _{ti}^+(\mathbf {w})+\gamma _{ti}^-(\mathbf {w})-\alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \right) \\ \vdots \\ \sum \limits _{t=1}^{m}\sum \limits _{i=1}^{l_t}x^n_{ti}\left( \left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{ti}-\left( \gamma _{ti}^+(\mathbf {w})+\gamma _{ti}^-(\mathbf {w})-\alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \right) \\ \end{bmatrix}\nonumber \\&\quad =z\mathbf {w}+2C\nonumber \\&\quad \begin{bmatrix} \sum \limits _{i=1}^{l}x^1_i\left( \left( \alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{i}-\left( \gamma _{i}^+(\mathbf {w})+\gamma _{i}^-(\mathbf {w})-\alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \right) \\ \vdots \\ \sum \limits _{i=1}^{t}x^n_i\left( \left( \alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \mathbf {w}^T\mathbf {x}_{i}-\left( \gamma _{i}^+(\mathbf {w})+\gamma _{i}^-(\mathbf {w})-\alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \right) \\ \end{bmatrix}\nonumber \\&\quad =z\mathbf {w}+2CX^T\left( A_w^TA_wX\mathbf {w}-A_w^T\mathbf {e}_w\right) \nonumber \\&\quad =(z-1)\mathbf {w}+\nabla f(\mathbf {w}). \end{aligned}$$

(44)

Therefore, it proves that:

$$\begin{aligned} \nabla f(\mathbf {w})=\sum \limits _{s=1}^{z}\nabla f^{(s)}(\mathbf {w})-(z-1)\mathbf {w}. \end{aligned}$$

Similarly, by plugging (39) in to $\sum _{s=1}^{z}\nabla ^2 f^{(s)}(\mathbf {w})\mathbf {v}$, we can get that

$$\begin{aligned} \sum \limits _{s=1}^{z}\nabla ^2 f^{(s)}(\mathbf {w})\mathbf {v}= & {} z\mathbf {v}+2C\sum \limits _{s=1}^{p}X^T_s\left( A^{(s)T}_wA^{(s)}_wX_s\mathbf {v}\right) \nonumber \\= & {} z\mathbf {v}+2C\nonumber \\&\begin{bmatrix} \sum \limits _{t=1}^{m}\sum \limits _{i=1}^{l_t}x^1_{ti}\left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {x}^T_{ti}\mathbf {v}-\left( \beta _{ti}^+(\mathbf {w},\mathbf {v})+\beta _{ti}^-(\mathbf {w},\mathbf {v})\right) \\ \vdots \\ \sum \limits _{t=1}^{m}\sum \limits _{i=1}^{l_t}x^n_{ti}\left( \alpha _{ti}^+(\mathbf {w})+\alpha _{ti}^-(\mathbf {w})\right) \mathbf {x}^T_{ti}\mathbf {v}-\left( \beta _{ti}^+(\mathbf {w},\mathbf {v})+\beta _{ti}^-(\mathbf {w},\mathbf {v})\right) \\ \end{bmatrix}\nonumber \\= & {} z\mathbf {v}+2C\nonumber \\&\begin{bmatrix} \sum \limits _{i=1}^{l}x^1_i\left( \alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \mathbf {x}^T_{i}\mathbf {v}-\left( \beta _{i}^+(\mathbf {w},\mathbf {v})+\beta _{i}^-(\mathbf {w},\mathbf {v})\right) \\ \vdots \\ \sum \limits _{i=1}^{l}x^n_i\left( \alpha _{i}^+(\mathbf {w})+\alpha _{i}^-(\mathbf {w})\right) \mathbf {x}^T_{i}\mathbf {v}-\left( \beta _{i}^+(\mathbf {w},\mathbf {v})+\beta _{i}^-(\mathbf {w},\mathbf {v})\right) \\ \end{bmatrix}\nonumber \\= & {} z\mathbf {v}+2CX^T\left( A_w^TA_wX\mathbf {v}\right) \nonumber \\= & {} (z-1)\mathbf {v}+\nabla ^2 f(\mathbf {w})\mathbf {v} \end{aligned}$$

(45)

which proves that

$$\begin{aligned} \nabla ^2 f(\mathbf {w})\mathbf {v}=\sum \limits _{s=1}^{z}\nabla ^2 f^{(s)}(\mathbf {w})\mathbf {v}-(z-1)\mathbf {v}. \end{aligned}$$

Consequently, Theorem 2 is proved. $\square$