Riemannian proximal gradient methods

Abstract

In the Euclidean setting the proximal gradient method and its accelerated variants are a class of efficient algorithms for optimization problems with decomposable objective. In this paper, we develop a Riemannian proximal gradient method (RPG) and its accelerated variant (ARPG) for similar problems but constrained on a manifold. The global convergence of RPG is established under mild assumptions, and the O(1/k) is also derived for RPG based on the notion of retraction convexity. If assuming the objective function obeys the Rimannian Kurdyka–Łojasiewicz (KL) property, it is further shown that the sequence generated by RPG converges to a single stationary point. As in the Euclidean setting, local convergence rate can be established if the objective function satisfies the Riemannian KL property with an exponent. Moreover, we show that the restriction of a semialgebraic function onto the Stiefel manifold satisfies the Riemannian KL property, which covers for example the well-known sparse PCA problem. Numerical experiments on random and synthetic data are conducted to test the performance of the proposed RPG and ARPG.

Proofs of Lemmas 6 and 7

1.1 Proof of Lemma 6

Proof

Since R is smooth and therefore $C^2$, the mapping $m: {{\,\mathrm{T}\,}}{\mathcal {M}} \times {\mathbb {R}} \rightarrow {{\,\mathrm{T}\,}}{\mathcal {M}}: (\eta , t) \mapsto \frac{D}{d t} \frac{d}{d t} R\left( t \eta \right)$ is continuous where $\frac{D}{d t}$ denotes the covariant derivative along the curve $t \mapsto R(t \eta )$, see definition of covariant derivative in e.g., [21, Proposition 2.2]. In addition, since the set ${\mathcal {D}} = \{ (\eta _x, t) \mid x \in {\bar{\varOmega }}, \Vert \eta _x\Vert _x = 1, 0 \le t \le \delta _T \}$ is compact, there exists a positive constant $b_2$ such that

$$\begin{aligned} \Vert m(\eta , t)\Vert \le b_2 \end{aligned}$$

(A.1)

for all $(\eta , t) \in {\mathcal {D}}$.

If $\eta _x = 0_x$, then the conclusion holds. Otherwise, let ${\tilde{\eta }}_x = \eta _x / \Vert \eta _x\Vert _x$. Since ${{\,\mathrm{dist}\,}}(x, y)$ is the shortest distance of a curve connecting x and y, we have

$$\begin{aligned} {{\,\mathrm{dist}\,}}(x, y) \le \int _0^{\Vert \eta _x\Vert _x} \left\| \frac{d}{d t} R_{x} \left( t {\tilde{\eta }}_x\right) \right\| _{R_x(t {\tilde{\eta }}_x)} d t, \end{aligned}$$

(A.2)

where the right side is the length of the curve $R_x(t \eta _x)$. Using the Cauchy-Schwarz inequality and the invariance of the metric by the Riemannian affine connection, we have

$$\begin{aligned} \left| \frac{d}{d t} \left\| \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x)\right\| \right|&= \left| \frac{d}{d t} \sqrt{ {\left\langle \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x),\frac{d}{d t} R_{x} (t {\tilde{\eta }}_x) \right\rangle _{}} } \right| = \left| \frac{ {\left\langle \frac{D}{d t} \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x) , \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x) \right\rangle _{}} }{ \left\| \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x) \right\| } \right| \\&\le \left\| \frac{D}{d t} \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x) \right\| \le b_2. \qquad ({\mathrm{by}}~(A.1)) \end{aligned}$$

It follows that

$$\begin{aligned} \int _0^{\Vert \eta _x\Vert _x} \left\| \frac{d}{d t} R_{x} (t {\tilde{\eta }}_x)\right\| _{R_x(t \eta _x)} d t\le & {} \int _0^{\Vert \eta _x\Vert _x} (1 + b_2 t) d t \nonumber \\= & {} \Vert \eta _x\Vert _x + \frac{b_2}{2} \Vert \eta _x\Vert _x^2 \le b_3 \Vert \eta _x\Vert _x, \end{aligned}$$

(A.3)

where $b_3 = 1 + b_2 \delta _T / 2$. Combining (A.2) and (A.3) yields the result. $\square$

1.2 Proof of Lemma 7

Proof

For any $x \in {\bar{\varOmega }}$, there exists a positive constant $\varrho _x$ and a neighborhood ${\mathcal {U}}_x$ of x such that ${\mathcal {U}}_x$ is a totally restrictive set with respect to $\varrho _x$. Since ${\bar{\varOmega }}$ is compact, there exists finite number of $x_i$ such that their totally restrictive sets covering ${\bar{\varOmega }}$, i.e., $\cup _{i = 1}^t {\mathcal {U}}_{x_i} \supset {\bar{\varOmega }}$. Let $\delta = \frac{1}{2} \min (\varrho _{x_i}, i = 1, \ldots , t)$. We have that for any $x \in {\bar{\varOmega }}$, the retraction R is a diffeomorphism on ${\mathbb {B}}(x, 2 \delta )$. Therefore, ${\mathcal {T}}_{R_{\eta _x}}^{\sharp }$ is invertible for any $\eta _x$ satisfying $\Vert \eta _x\Vert _x < 2 \delta$.

Since ${\mathcal {T}}_{R_{\eta _x}}^{-\sharp }$ is smooth with respect to $\eta _x$ and the set $\{\eta _x \mid x \in {{\bar{\varOmega }}}, \Vert \eta _x\Vert \le \delta \}$ is compact, there exists a constant $L_t > 0$ such that

$$\begin{aligned} \Vert {\mathcal {T}}_{R_{\eta _x}}^{-\sharp }\Vert \le L_t, \forall \eta _x \in \{\eta _x \mid x \in {{\bar{\varOmega }}}, \Vert \eta _x\Vert \le \delta \}. \end{aligned}$$

(A.4)

By Lemma 6, there exists a positive constant $\kappa$ such that

$$\begin{aligned} {{\,\mathrm{dist}\,}}(x, R_x(\eta _x)) \le \kappa \Vert \eta _x\Vert _x \end{aligned}$$

(A.5)

for all $x \in {\bar{\varOmega }}$ and for all $\eta _x \in {\mathcal {B}}(0_x, \delta )$. Let ${\tilde{\delta }} = \min (\delta , i({\bar{\varOmega }}) / \kappa )$. For all $\eta _x \in {\mathcal {B}}(0_x, {\tilde{\delta }})$ it holds that

$$\begin{aligned} {{\,\mathrm{dist}\,}}(x, R_x(\eta _x)) \le \kappa \Vert \eta _x\Vert _x \le i({\bar{\varOmega }}). \end{aligned}$$

(A.6)

By the definition of locally Lipschitz continuity of a vector field, we have $\Vert {\mathcal {P}}_{\gamma }^{0 \leftarrow 1} \xi _y - \xi _x\Vert _x \le L_v {{\,\mathrm{dist}\,}}(x, y)$ for any $x, y \in {{\bar{\varOmega }}}$ and ${{\,\mathrm{dist}\,}}(x, y) < i({\bar{\varOmega }})$. Since the parallel translation is isometric, it holds that $\Vert \xi _y - {\mathcal {P}}_{\gamma }^{1 \leftarrow 0} \xi _x\Vert _y \le L_v {{\,\mathrm{dist}\,}}(x, y)$. Using (A.5) and (A.6) yields

$$\begin{aligned} \Vert \xi _y - {\mathcal {P}}_{\gamma }^{1 \leftarrow 0} \xi _x\Vert _x \le L_v {{\,\mathrm{dist}\,}}(x, y) \le L_v \kappa \Vert \eta _x\Vert _x, \end{aligned}$$

(A.7)

for all $\eta _x \in {\mathcal {B}}(0_x, {\tilde{\delta }})$, where $y = R_x(\eta _x)$.

By [32, Lemma 3.5], for any ${\bar{x}} \in {\mathcal {M}}$, there exists a neighborhood ${\mathcal {U}}_{{\bar{x}}}$ of ${\bar{x}}$ and a positive number $L_{{\bar{x}}}$ such that for all $x, y \in {\mathcal {U}}_{{\bar{x}}}$ it holds that

$$\begin{aligned} \Vert {\mathcal {P}}_{\gamma }^{1 \rightarrow 0} \xi _x - {\mathcal {T}}_{\eta _x}^{-\sharp } \xi _x\Vert _y \le L_x \Vert \xi _x\Vert _x \Vert \eta _x\Vert _x. \end{aligned}$$

Since ${{\bar{\varOmega }}}$ is compact, there exist finite number of ${\bar{x}}$, denoted by ${\bar{x}}_1, \ldots , {\bar{x}}_t$, such that $\cup _{i=1}^t {\mathcal {U}}_{{\bar{x}}_i} \supset \varOmega$. Let $L_{cc}$ denote $\max (L_{{\bar{x}}_i}, i = 1, \ldots , t)$, and $\sigma = \sup _{r}\left\{ r \in {\mathbb {R}} \mid \exists i, \hbox { such that } {\mathbb {B}}(z, r) \subseteq {\mathcal {U}}_{{\bar{x}}_i} \forall z \in {{\bar{\varOmega }}} \right\}$. Since the number of ${\bar{x}}_i$ is finite, we have $L_{cc} < \infty$ and $\sigma > 0$. Therefore, for any $x, y \in {{\bar{\varOmega }}}$ satisfying ${{\,\mathrm{dist}\,}}(x, y) < \sigma$, it holds that

$$\begin{aligned} \Vert {\mathcal {P}}_{\gamma }^{1 \rightarrow 0} \xi _x - {\mathcal {T}}_{\eta _x}^{-\sharp } \xi _x\Vert _y \le L_{cc} \Vert \xi _x\Vert _x \Vert \eta _x\Vert _x. \end{aligned}$$

(A.8)

Note that $\Vert \eta _x\Vert _x < \sigma / \kappa$ implies ${{\,\mathrm{dist}\,}}(x, y) < \sigma$ by (A.5). It follows from (A.4), (A.7) and (A.8) that for any $x, y \in {{\bar{\varOmega }}}$ satisfying $\Vert \eta _x\Vert _x < \min (\sigma / \kappa , {\tilde{\delta }})$,

$$\begin{aligned} \Vert \xi _y - {\mathcal {T}}_{\eta _x}^{-\sharp } (\xi _x + a \eta _x)\Vert _y\le & {} \Vert \xi _y - {\mathcal {P}}_{\gamma }^{1 \leftarrow 0} \xi _x\Vert _y + \Vert {\mathcal {P}}_{\gamma }^{1 \leftarrow 0} \xi _x - {\mathcal {T}}_{\eta _x}^{-\sharp } \xi _x\Vert _y + \Vert {\mathcal {T}}_{\eta _x}^{-\sharp } a \eta _x\Vert _y \\\le & {} L_c \Vert \eta _x\Vert _x, \end{aligned}$$

where $L_{c} = L_v \kappa + L_{cc} \sup _{x \in {{\bar{\varOmega }}}} \Vert \xi _x\Vert _x + a L_t$.$\square$