An accelerated optimization algorithm for the elastic-net extreme learning machine

Abstract

Extreme learning machine (ELM) has received considerable attention due to its rapid learning speed and powerful fitting capabilities. One of its important variants, the elastic-net ELM (Enet-ELM), was recently proposed to improve its sparsity and stability of simulations simultaneously. However, entering the era of big data, the explosive growth of data volume and dimensions poses a huge challenge to Enet-ELM. On the other hand, the alternating direction method of multipliers (ADMM) is a powerful iterative algorithm for solving large-scale optimization problems by splitting a large problem into a set of executable sub-problems. But its performance is highly restricted by its astringency and convergence rates. In this paper, we therefore develop a novel Enet-ELM algorithm based on the over-relaxed ADMM, termed over-relaxed Enet-ELM (OE-ELM), which accelerates model training by applying the results of the previous iteration to the next iteration. Besides, we also propose a parallel version of OE-ELM (POE-ELM) to implement parallel and distributed computation, which is trained by the consensus over relaxation ADMM algorithm. Finally, the convergence analysis conducted on the two proposed algorithms proves the effectiveness of model training, and extensive experiments on classification and regression datasets demonstrate their competitiveness in accuracy and convergence rate.

Appendix A Proof of Theorem 1

Proof

Note that POE-ELM is equivalent to the iterations of Eqs. (37)–(41), so we just need to prove that the iterations of Eqs. (37)–(41) converge to the global optimal solution because for $\forall i \ne j$ there must be $\widetilde{{\textbf {B}}}_i^\mathsf {T}\widetilde{{\textbf {B}}}_j = {\textbf {0}}$ ($i, j = 1, \ldots , M$), both $\widetilde{{\textbf {A}}}$ and $\widetilde{{\textbf {C}}}$ have full column rank, and $\widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {C}}} = {\textbf {0}}$. By the first-order optimality conditions of Eqs. (37)–(41), we derive that

$$\begin{aligned}&\theta _j({\varvec{\beta }}_j) - \theta _j({\varvec{\beta }}_j^{k+1}) + ({\varvec{\beta }}_j \nonumber \\&\quad - {\varvec{\beta }}_j^{k+1})^\mathsf {T} \Bigg [ -\widetilde{{\textbf {B}}}_j^\mathsf {T} {\textbf {u}}^k + \rho \widetilde{{\textbf {B}}}_j^\mathsf {T} (\sum _{l=1}^{j}\widetilde{{\textbf {B}}}_l{\varvec{\beta }}_l^{k+1} \nonumber \\&\quad + \sum ^{M}_{l=j+1}\widetilde{{\textbf {B}}}_l{\varvec{\beta }}_l^k + \widetilde{{\textbf {A}}}{} {\textbf {a}}^k \nonumber \\&\quad + \widetilde{{\textbf {C}}}{} {\textbf {c}}^k + \widetilde{{\textbf {d}}}) \Bigg ] \ge 0, \ \forall {\varvec{\beta }}_j \in \mathbb {R}^L, j = 1, \ldots , M, \end{aligned}$$

(A1)

$$\begin{aligned}&\theta _{M+1}({\textbf {a}}) - \theta _{M+1}({\textbf {a}}^{k+1}) + ({\textbf {a}} - {\textbf {a}}^{k+1})^\mathsf {T} \Bigg \{ -\widetilde{{\textbf {A}}}^\mathsf {T} {\textbf {u}}^k \nonumber \\&\quad + \rho \widetilde{{\textbf {A}}}^\mathsf {T} [\alpha \sum _{j=1}^M\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} \nonumber \\&\quad + \widetilde{{\textbf {A}}} ({\textbf {a}}^{k+1} - (1 - \alpha ){\textbf {a}}^k) + \alpha \widetilde{{\textbf {C}}}{} {\textbf {c}}^k \nonumber \\&\quad + \alpha \widetilde{{\textbf {d}}}] \Bigg \} \ge 0, \forall {\textbf {a}} \in \mathbb {R}^N, \end{aligned}$$

(A2)

$$\begin{aligned}&\theta _{M+2}({\textbf {c}}) - \theta _{M+2}({\textbf {c}}^{k+1}) + ({\textbf {c}} - {\textbf {c}}^{k+1})^\mathsf {T} \Bigg \{ -\widetilde{{\textbf {C}}}^\mathsf {T} {\textbf {u}}^k \nonumber \\&\quad + \rho \widetilde{{\textbf {C}}}^\mathsf {T} [\alpha \sum _{j=1}^M\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} \nonumber \\&\quad + \widetilde{{\textbf {A}}} ({\textbf {a}}^{k+1} - (1 - \alpha ){\textbf {a}}^k) + \widetilde{{\textbf {C}}} ({\textbf {c}}^{k+1} - (1 - \alpha ){\textbf {c}}^k) \nonumber \\&\quad + \alpha \widetilde{{\textbf {d}}}] \Bigg \} \ge 0, \forall {\textbf {c}} \in \mathbb {R}^{L}, \end{aligned}$$

(A3)

$$\begin{aligned}&{\textbf {u}}^{k+1} = {\textbf {u}}^k - \rho \big [\alpha \sum _{j=1}^M\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} + \widetilde{{\textbf {A}}} ({\textbf {a}}^{k+1} \nonumber \\&\quad - (1 - \alpha ){\textbf {a}}^k) + \widetilde{{\textbf {C}}} ({\textbf {c}}^{k+1} \nonumber \\&\quad - (1 - \alpha ){\textbf {c}}^k) + \alpha \widetilde{{\textbf {d}}}\big ], \end{aligned}$$

(A4)

where $\theta _j({\varvec{\beta }}_j) = \lambda (1 - \sigma )\Vert {\varvec{\beta }}_j \Vert ^2_2$, $\theta _{M+1}({\textbf {a}}) = \sum ^M_{j = 1} \Vert {\textbf {a}}_j \Vert ^2_2$, $\theta _{M+2}({\textbf {c}}) = \lambda \sigma \Vert {\textbf {c}} \Vert _1,$ and ${\textbf {u}} = -\rho [{\textbf {v}}_{11}^\mathsf {T}, \ldots , {\textbf {v}}_{1M}^\mathsf {T}, {\textbf {v}}_{21}^\mathsf {T}, \ldots , {\textbf {v}}_{2M}^\mathsf {T}]^\mathsf {T}$.

For simplicity, we also define

$$\begin{aligned}&{\textbf {p}}^k := \begin{pmatrix} {\varvec{\beta }}_1^k \\ \vdots \\ {\varvec{\beta }}_M^k \\ {\textbf {a}}^k \\ {\textbf {c}}^k \end{pmatrix}, \ {\textbf {q}}^k := \begin{pmatrix} {\varvec{\beta }}_1^k \\ \vdots \\ {\varvec{\beta }}_M^k \\ {\textbf {a}}^k \\ {\textbf {c}}^k \\ {\textbf {u}}^k \end{pmatrix}, \ {\textbf {r}}^k := \begin{pmatrix} {\textbf {a}}^k\\ {\textbf {c}}^k\\ {\textbf {u}}^k \end{pmatrix}, \end{aligned}$$

(A5)

$$\begin{aligned}&\widetilde{{\varvec{\beta }}}_j^k := {\varvec{\beta }}_j^{k+1}, \ \widetilde{{\textbf {a}}}^k := {\textbf {a}}^{k+1}, \ \widetilde{{\textbf {c}}}^k := {\textbf {c}}^{k+1}, \end{aligned}$$

(A6)

$$\begin{aligned}&\widetilde{{\textbf {u}}}^k := {\textbf {u}}^k - \rho (\sum ^M_{j=1}\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} + \widetilde{{\textbf {A}}}{} {\textbf {a}}^k + \widetilde{{\textbf {C}}}{} {\textbf {c}}^k + \widetilde{{\textbf {d}}}), \end{aligned}$$

(A7)

$$\begin{aligned}&\widetilde{{\textbf {p}}}^k := \begin{pmatrix} \widetilde{{\varvec{\beta }}}_1^k\\ \vdots \\ \widetilde{{\varvec{\beta }}}_M^k \\ \widetilde{{\textbf {a}}}^k \\ \widetilde{{\textbf {c}}}^k \end{pmatrix}, \ \widetilde{{\textbf {q}}}^k := \begin{pmatrix} \widetilde{{\varvec{\beta }}}_1^k \\ \vdots \\ \widetilde{{\varvec{\beta }}}_M^k \\ \widetilde{{\textbf {a}}}^k \\ \widetilde{{\textbf {c}}}^k \\ \widetilde{{\textbf {u}}}^k \end{pmatrix}, \ \widetilde{{\textbf {r}}}^k := \begin{pmatrix} \widetilde{{\textbf {a}}}^k \\ \widetilde{{\textbf {c}}}^k \\ \widetilde{{\textbf {u}}}^k \end{pmatrix}, \end{aligned}$$

(A8)

where (${\varvec{\beta }}_1^{k+1}, \ldots , {\varvec{\beta }}_M^{k+1}, {\textbf {a}}^{k+1}, {\textbf {c}}^{k+1}$) is generated by Eqs. (37)-(41) from given (${\textbf {a}}^k, {\textbf {c}}^k, {\textbf {u}}^k$). Then Eq. (A4) can be reformulated as

$$\begin{aligned} {\textbf {u}}^{k+1} = \&\widetilde{{\textbf {u}}}^k + \rho [\widetilde{{\textbf {A}}} ({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) + \widetilde{{\textbf {C}}} ({\textbf {c}}^k - \widetilde{{\textbf {c}}}^k)] + (1 - \alpha )({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k). \end{aligned}$$

(A9)

Substituting the definition of $\widetilde{{\textbf {u}}}^k$ and Eq. (A9) into Eqs. (A1)-(A4), we have

$$\begin{aligned}&\theta _j({\varvec{\beta }}_j) - \theta _j({\varvec{\beta }}_j^{k+1}) + ({\varvec{\beta }}_j \nonumber \\&\quad - {\varvec{\beta }}_j^{k+1})^\mathsf {T} \Bigg \{ -\widetilde{{\textbf {B}}}_j^\mathsf {T}\widetilde{{\textbf {u}}}^k - \rho \widetilde{{\textbf {B}}}_j^\mathsf {T} [\sum ^M_{l = j + 1} \widetilde{{\textbf {B}}}_l({\varvec{\beta }}_l^{k+1} - {\varvec{\beta }}_l^k)] \Bigg \} \nonumber \\&\quad \ge 0, \ \forall {\varvec{\beta }}_j \in \mathbb {R}^L, \end{aligned}$$

(A10)

$$\begin{aligned}&\theta _{M+1}({\textbf {a}}) - \theta _{M+1}({\textbf {a}}^{k+1}) \nonumber \\&\quad + ({\textbf {a}} - {\textbf {a}}^{k+1})^\mathsf {T} \Bigg \{ -\widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {u}}}^k \nonumber \\&\quad - \rho \widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {A}}}({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) \nonumber \\&\quad - (1 - \alpha )\widetilde{{\textbf {A}}}^\mathsf {T}({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k) \Bigg \} \ge 0, \ \forall {\textbf {a}} \in \mathbb {R}^N, \end{aligned}$$

(A11)

$$\begin{aligned}&\theta _{M+2}({\textbf {c}}) - \theta _{M+2}({\textbf {c}}^{k+1}) \nonumber \\&\quad + ({\textbf {c}} - {\textbf {c}}^{k+1})^\mathsf {T} \Bigg \{ -\widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {u}}}^k - \rho \widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {A}}}({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) \nonumber \\&\quad - \rho \widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {C}}}({\textbf {c}}^k - \widetilde{{\textbf {c}}}^k) \nonumber \\&\quad - (1 - \alpha )\widetilde{{\textbf {C}}}^\mathsf {T}({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k) \Bigg \} \ge 0, \ \forall {\textbf {c}} \in \mathbb {R}^{L}, \end{aligned}$$

(A12)

$$\begin{aligned}&(\sum ^M_{j=1}\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} + \widetilde{{\textbf {A}}}{} {\textbf {a}}^{k+1} + \widetilde{{\textbf {C}}}{} {\textbf {c}}^{k+1} \nonumber \\&\quad + \widetilde{{\textbf {d}}}) + \widetilde{{\textbf {A}}} ({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) + \widetilde{{\textbf {C}}} ({\textbf {c}}^k - \widetilde{{\textbf {c}}}^k) \nonumber \\&\quad - \frac{1}{\rho }({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k)= {\textbf {0}}. \end{aligned}$$

(A13)

By Eq. (A13), we derive that

$$\begin{aligned}&({\textbf {u}} - \widetilde{{\textbf {u}}}^k)^\mathsf {T}[(\sum ^M_{j=1}\widetilde{{\textbf {B}}}_j{\varvec{\beta }}_j^{k+1} + \widetilde{{\textbf {A}}}{} {\textbf {a}}^{k+1} \nonumber \\&\quad + \widetilde{{\textbf {C}}}{} {\textbf {c}}^{k+1} + \widetilde{{\textbf {d}}}) + \widetilde{{\textbf {A}}} ({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) + \widetilde{{\textbf {C}}} ({\textbf {c}}^k - \widetilde{{\textbf {c}}}^k) \nonumber \\&\quad - \frac{1}{\rho }({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k)] = 0, \ \forall {\textbf {u}} \in \mathbb {R}^{N+ML}. \end{aligned}$$

(A14)

Thus, by adding up Eqs. (A10)–(A12) and (A14), since $\widetilde{{\textbf {B}}}_i^\mathsf {T}\widetilde{{\textbf {B}}}_j = {\textbf {0}}$ ($\forall i \ne j$, $i, j = 1, \ldots , M$) and $\widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {C}}} = {\textbf {0}}$, we can deduce the following inequality:

$$\begin{aligned} \theta ({\textbf {p}}) - \theta (\widetilde{{\textbf {p}}}^k) + ({\textbf {q}} - \widetilde{{\textbf {q}}}^k)^\mathsf {T}[{\mathrm{F}}(\widetilde{{\textbf {q}}}^k) - {\textbf {Q}}_0({\textbf {q}}^k - \widetilde{{\textbf {q}}}^k)] \ge 0, \end{aligned}$$

(A15)

where

$$\begin{aligned} {\textbf {Q}}_0 = \begin{pmatrix} {\textbf {0}}&{}{\textbf {0}}&{}{\textbf {0}}&{}{\textbf {0}} \\ {\textbf {0}}&{}\rho \widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {A}}}&{}{\textbf {0}}&{}(1 - \alpha )\widetilde{{\textbf {A}}}^\mathsf {T} \\ {\textbf {0}}&{}{\textbf {0}}&{}\rho \widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {C}}}&{}(1 - \alpha )\widetilde{{\textbf {C}}}^\mathsf {T} \\ {\textbf {0}}&{}-\widetilde{{\textbf {A}}}&{}-\widetilde{{\textbf {C}}}&{}\frac{1}{\rho }{} {\textbf {I}}_{N+ML} \end{pmatrix}. \end{aligned}$$

(A16)

By the definition of ${\textbf {q}}$, ${\textbf {q}}^k$, $\widetilde{{\textbf {q}}}^k$ and ${\textbf {r}}$, ${\textbf {r}}^k$, $\widetilde{{\textbf {r}}}^k$, Eq. (A15) can be simplified as

$$\begin{aligned} \theta ({\textbf {p}}) - \theta (\widetilde{{\textbf {p}}}^k) + ({\textbf {q}} - \widetilde{{\textbf {q}}}^k)^\mathsf {T}{\mathrm{F}}(\widetilde{{\textbf {q}}}^k) \ge ({\textbf {r}} - \widetilde{{\textbf {r}}}^k)^\mathsf {T}{} {\textbf {Q}}({\textbf {r}}^k - \widetilde{{\textbf {r}}}^k), \end{aligned}$$

(A17)

where

$$\begin{aligned} {\textbf {Q}} = \begin{pmatrix} \rho \widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {A}}}&{}{\textbf {0}}&{}(1 - \alpha )\widetilde{{\textbf {A}}}^\mathsf {T} \\ {\textbf {0}}&{}\rho \widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {C}}}&{}(1 - \alpha )\widetilde{{\textbf {C}}}^\mathsf {T} \\ -\widetilde{{\textbf {A}}}&{}-\widetilde{{\textbf {C}}}&{}\frac{1}{\rho }{} {\textbf {I}}_{N+ML} \end{pmatrix}. \end{aligned}$$

(A18)

According to the full column rank of $\widetilde{{\textbf {A}}}$ and $\widetilde{{\textbf {C}}}$, it is obvious that

$$\begin{aligned} {\textbf {Q}}^\mathsf {T} + {\textbf {Q}} = \begin{pmatrix} 2\rho \widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {A}}}&{}{\textbf {0}}&{}- \alpha \widetilde{{\textbf {A}}}^\mathsf {T} \\ {\textbf {0}}&{}2\rho \widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {C}}}&{}- \alpha \widetilde{{\textbf {C}}}^\mathsf {T} \\ - \alpha \widetilde{{\textbf {A}}}&{}- \alpha \widetilde{{\textbf {C}}}&{}\frac{2}{\rho }{} {\textbf {I}}_{N+ML} \end{pmatrix} \end{aligned}$$

(A19)

is a positive definite matrix. Then, since Eq. (A9) has the expression that

$$\begin{aligned} {\textbf {u}}^{k+1} = \ {\textbf {u}}^k + \rho [\widetilde{{\textbf {A}}}({\textbf {a}}^k - \widetilde{{\textbf {a}}}^k) + \widetilde{{\textbf {C}}}({\textbf {c}}^k - \widetilde{{\textbf {c}}}^k)] - \alpha ({\textbf {u}}^k - \widetilde{{\textbf {u}}}^k), \end{aligned}$$

(A20)

the following equation holds:

$$\begin{aligned} {\textbf {r}}^{k+1} = {\textbf {r}}^k - \gamma {\textbf {M}}({\textbf {r}}^k - \widetilde{{\textbf {r}}}^k), \end{aligned}$$

(A21)

where $\gamma = 1$ and

$$\begin{aligned} {\textbf {M}} = \begin{pmatrix} {\textbf {I}}_N&{}{\textbf {0}}&{}{\textbf {0}} \\ {\textbf {0}}&{}{\textbf {I}}_L&{}{\textbf {0}} \\ - \rho \widetilde{{\textbf {A}}}&{}- \rho \widetilde{{\textbf {C}}}&{}\alpha {\textbf {I}}_{N+ML} \end{pmatrix}. \end{aligned}$$

(A22)

Note that ${\textbf {M}}$ is a nonsingular matrix whose inverse matrix is

$$\begin{aligned} {\textbf {M}}^{-1} = \begin{pmatrix} {\textbf {I}}_N&{}{\textbf {0}}&{}{\textbf {0}} \\ {\textbf {0}}&{}{\textbf {I}}_L&{}{\textbf {0}} \\ \frac{\rho }{\alpha }\widetilde{{\textbf {A}}}&{}\frac{\rho }{\alpha }\widetilde{{\textbf {C}}}&{}\frac{1}{\alpha }{} {\textbf {I}}_{N+ML} \end{pmatrix}. \end{aligned}$$

(A23)

Furthermore, we can define a symmetric and positive definite matrix as

$$\begin{aligned} {\textbf {P}} := {\textbf {Q}}{} {\textbf {M}}^{-1} = \begin{pmatrix} \frac{\rho }{\alpha }\widetilde{{\textbf {A}}}^\mathsf {T}\widetilde{{\textbf {A}}}&{}{\textbf {0}}&{}\frac{1 - \alpha }{\alpha }\widetilde{{\textbf {A}}}^\mathsf {T} \\ {\textbf {0}}&{}\frac{\rho }{\alpha }\widetilde{{\textbf {C}}}^\mathsf {T}\widetilde{{\textbf {C}}}&{}\frac{1 - \alpha }{\alpha }\widetilde{{\textbf {C}}}^\mathsf {T} \\ \frac{1 - \alpha }{\alpha }\widetilde{{\textbf {A}}}&{}\frac{1 - \alpha }{\alpha }\widetilde{{\textbf {C}}}&{}\frac{1}{\alpha \rho }{} {\textbf {I}}_{N+ML} \end{pmatrix}. \end{aligned}$$

(A24)

Consequently, we find that

$$\begin{aligned} {\textbf {G}} := {\textbf {Q}}^\mathsf {T} + {\textbf {Q}} - \gamma {\textbf {M}}^\mathsf {T}{} {\textbf {P}}{} {\textbf {M}} = \begin{pmatrix} {\textbf {0}}&{}{\textbf {0}}&{}{\textbf {0}} \\ {\textbf {0}}&{}{\textbf {0}}&{}{\textbf {0}} \\ {\textbf {0}}&{}{\textbf {0}}&{}\frac{2 - \alpha }{\rho }{} {\textbf {I}}_{N+ML} \end{pmatrix} \end{aligned}$$

(A25)

is obviously positive semidefinite.

Finally, the matrices ${\textbf {Q}}^\mathsf {T} + {\textbf {Q}}$, ${\textbf {P}}$ and ${\textbf {G}}$ derived from (A19), (A24) and (A25) satisfy the convergence condition proposed in Lemma 1. As a result, the iterations of Eqs. (37)–(41), namely POE-ELM, converge to the global optimal solution. $\square$