Sparse group fused lasso for model segmentation: a hybrid approach

Abstract

This article introduces the sparse group fused lasso (SGFL) as a statistical framework for segmenting sparse regression models with multivariate time series. To compute solutions of the SGFL, a nonsmooth and nonseparable convex program, we develop a hybrid optimization method that is fast, requires no tuning parameter selection, and is guaranteed to converge to a global minimizer. In numerical experiments, the hybrid method compares favorably to state-of-the-art techniques with respect to computation time and numerical accuracy; benefits are particularly substantial in high dimension. The method’s statistical performance is satisfactory in recovering nonzero regression coefficients and excellent in change point detection. An application to air quality data is presented. The hybrid method is implemented in the R package sparseGFL available on the author’s Github page.

Appendices

Proof of Theorem 1

Recall the notations of Sect. 3.2:

$$\begin{aligned} \begin{aligned} g(\beta _t)&= \lambda _1 \Vert \beta _t \Vert _1 + \lambda _2 w_{t-1}\Vert \beta _t - \hat{\beta }_{t-1} \Vert _2 + \lambda _2 w_t \Vert \beta _t - \hat{\beta }_{t+1} \Vert _2, \\ g_1(\beta _t)&= \lambda _1 \Vert \beta _t \Vert _1 , \\ g_2(\beta _t)&= \lambda _2 w_{t-1} \Vert \beta _t - \hat{\beta }_{t-1} \Vert _2 +\lambda _2w_t \Vert \beta _t- \hat{\beta }_{t+1} \Vert _2 + (L_t/2) \Vert \beta _t -z_t \Vert _2^2 ,\\ \bar{g} (\beta _t)&= g(\beta _t) + (L_t/2) \left\| \beta _t - z_t \right\| _2^2 = g_1 (\beta _t)+ g_2 (\beta _t) , \\ \quad \gamma _n&= \big ( L_t + (\lambda _2 w_{t-1} / \Vert \beta _t^{n} - \hat{\beta }_{t-1} \Vert _2 ) + (\lambda _2 w_t / \Vert \beta _{t}^{n} - \hat{\beta }_{t+1} \Vert _2 ) \big )^{-1}, \\ \beta _t^*&= \mathrm {argmin}\, \bar{g} = \mathrm {prox}_{g/L_t}(z_t) , \quad r_n = \bar{g} (\beta _t^n) - \bar{g} (\beta _t^*). \end{aligned} \end{aligned}$$

In view of Remark 2 framing the iterative soft-thresholding scheme (23)–(24) as a proximal gradient method, we can establish the linear convergence of this scheme to $\mathrm {prox}_{g/L_t}(z_t)$ by adapting the results of Bredies and Lorenz (2008) to a nonsmooth setting. Essentially, the proof of linear convergence in Bredies and Lorenz (2008) works by first establishing a lower bound on $\bar{g}(\beta _t^{n})-\bar{g}(\beta _t^n)$, the decrease in the objective function between successive iterations of the proximal gradient method (Lemma 1). This general result shows in particular that when using sufficiently small step sizes, the proximal gradient is a descent method. After that, under the additional assumptions that $g_2$ is convex and that $\Vert \beta _t^n - \beta _t^*\Vert _2^2 \le c r_n$ for some $c>0$, the lower bound of Lemma 1 is exploited to show the exponential decay of $(r_n)$ and the linear convergence of $(\beta _t^n)$ (Proposition 2). In a third movement, the lower bound of Lemma 1 is decomposed as a Bregman-like distance term involving $g_1$ plus a Taylor remainder term involving $g_2$. The specific nature of $g_1$ ($\ell _1$ norm) and possible additional regularity conditions on $g_2$ (typically, strong convexity) are then used to establish the linear convergence result (Theorem 2). For brevity, we refer the reader to Bredies and Lorenz (2008) for the exact statement of these results.

Lemma 1 and Proposition 2 of Bredies and Lorenz (2008) posit, among other things, that the “smooth” part of the objective, $g_2$ in our notations, is differentiable everywhere and has a Lipschitz-continuous gradient. In the present case, $g_2$ is not differentiable at $\hat{\beta }_{t-1}$ and $\hat{\beta }_{t+1}$; however it is differentiable everywhere else and its gradient is Lipschitz-continuous in a local sense. The main effort required for us is to show that Lemma 1 still holds if the points of nondifferentiability of $g_2$ are not on segments joining the iterates $\beta _t^n , n\ge 0$. Put differently, the iterative soft-thresholding scheme should not cross $\hat{\beta }_{t-1}$ and $\hat{\beta }_{t+1}$ on its path. This is where the requirement that $\bar{g}(\beta _t^0) < \min ( \bar{g}(\hat{\beta }_{t-1}), \bar{g}(\hat{\beta }_{t+1}))$ in Theorem 1 plays a crucial part. We now proceed to adapt Lemma 1, after which we will establish the premises of Theorem 2 of Bredies and Lorenz (2008).

Adaptation of Lemma 1of Bredies and Lorenz (2008). The main result we need prove is that

$$\begin{aligned} \forall n\in \mathbb {N}, \quad \big \{ \hat{\beta }_{t-1} , \hat{\beta }_{t+1} \big \} \bigcap ^{\,} \left\{ \alpha \beta _{t}^{n} + (1-\alpha ) \beta _{t}^{n+1} : 0 \le \alpha \le 1 \right\} = \emptyset \,. \end{aligned}$$

(40)

Once this is established, we may follow the proof of Proposition 2 without modification. In particular, we will be in position to state that

$$\begin{aligned} \left\| \nabla g_2(\beta _t^n + \alpha (\beta _t^{n+1} - \beta _t^n ) ) - \nabla g_2(\beta _t^{n})\right\| _2 \le \alpha \tilde{L}_n \left\| \beta _t^{n+1} - \beta _t^{n} \right\| _2 \end{aligned}$$

(41)

for all $n\in \mathbb {N}$ and $\alpha \in [0,1]$, where

$$\begin{aligned} \tilde{L}_n = L_t + \frac{2\lambda _2 w_{t-1} }{ \Vert \beta _t^n - \hat{\beta }_{t-1} \Vert _2 } +\frac{2\lambda _2 w_t }{ \Vert \beta _t^n - \hat{\beta }_{t+1} \Vert _2 } . \end{aligned}$$

Note that the left-hand side in (41) is not well defined if (40) does not hold. Combining the local Lipschitz property (41) with the step size condition $\gamma _{n} < 2 / \tilde{L}_n$, we may go on to establish the descent property (3.5) of Bredies and Lorenz (2008):

$$\begin{aligned} \bar{g}(\beta _t^{n+1}) \le \bar{g}(\beta _t^{n}) - \delta D_{\gamma _n}(\beta _t^{n}) \end{aligned}$$

(42)

where

$$\begin{aligned} D_{\gamma _n}(\beta _t^{n}) = g_1(\beta _t^{n}) - g_1(\beta _t^{n+1}) + \nabla g_2(\beta _t^{n})'(\beta _t^{n}-\beta _t^{n+1}) \quad \text { and } \quad \delta = 1 - \frac{\max _{n} \gamma _n \tilde{L}_n }{ 2} . \end{aligned}$$

Lemma 1 shows that $D_{\gamma _n}(\beta _t^{n}) \ge \Vert \beta _t^{n}-\beta _t^{n+1}\Vert _2^2 / \gamma _n \ge 0$. To show the positivity of $\delta$, note that

$$\begin{aligned} \gamma _n \tilde{L}_n = 2 - L_t \left( \displaystyle L_t + \frac{\lambda _2 w_{t-1} }{ \Vert \beta _t^{n} - \hat{\beta }_{t-1}\Vert _2} + \frac{\lambda _2 w_{t}}{ \Vert \beta _t^{n} - \hat{\beta }_{t+1}\Vert _2 } \right) ^{-1} . \end{aligned}$$

Given the descent property of $(\beta _n)$ for $\bar{g}$, the assumption $\bar{g}(\beta _t^0) < \min (\bar{g}(\hat{\beta }_{t-1}),\bar{g}(\hat{\beta }_{t+1}))$, and the convexity of the sublevel sets of $\bar{g}$, it holds that $\Vert \beta _t^{n} - \hat{\beta }_{t-1}\Vert _2 \ge d(\hat{\beta }_{t-1} , \{ \beta _t : \bar{g}(\beta _t)\le \bar{g}(\beta _t^0) \})$ for all $n\in \mathbb {N}$; an analog inequality holds for $\hat{\beta }_{t+1}$. Denoting these positive lower bounds by $m_{t-1}$ and $m_{t+1}$, we have

$$\begin{aligned} 0 < \frac{1}{2} \left( \displaystyle 1 + \frac{\lambda _2 w_{t-1} }{L_t m_{t-1}} + \frac{\lambda _2 w_{t}}{ L_t m_{t+1} } \right) ^{-1} \le \delta \le \frac{1}{2} . \end{aligned}$$

(43)

Together, the step size condition $\gamma _{n} < 2 / \tilde{L}_n$, descent property (42), and lower bound (43) finish to establish Lemma 1 and the precondition of Proposition 2 of Bredies and Lorenz (2008).

It remains to prove (40). We will show a weaker form of (42), namely that $\bar{g}(\beta _t^{n+1}) \le \bar{g}(\beta _t^{n})$ for all n. This inequality, combined with the convexity of $\bar{g}$ and the assumption $\bar{g}(\beta _t^0) < \min (\bar{g}(\hat{\beta }_{t-1}),\bar{g}(\hat{\beta }_{t+1}))$, implies that $\hat{\beta }_{t-1}$ and $\hat{\beta }_{t+1}$ cannot be on a segment joining $\beta _t^{n}$ and $\beta _t^{n+1}$. Otherwise, the convexity of $\bar{g}$ would imply that, say, $\bar{g}(\hat{\beta }_{t-1}) \le \max ( \bar{g}(\beta _t^{n}), \bar{g}(\beta _t^{n+1})) \le \bar{g}(\beta _t^{n})\le \cdots \le \bar{g}(\beta _t^{0}) < \bar{g}(\hat{\beta }_{t-1})$, a contradiction.

To prove the simple descent property, we start with an easy lemma stated without proof.

Lemma 1

For all $x,y \in \mathbb {R}^p$ such that $y \ne 0_p$,

$$\begin{aligned} \Vert x \Vert _2 \le \Vert y \Vert _2 + \frac{ y'(x-y) }{\Vert y \Vert _2 } + \frac{\Vert x - y\Vert _2^2 }{2\Vert y\Vert _2} \,. \end{aligned}$$

Applying this lemma to $x= \beta _t - \hat{\beta }_{t \pm 1}$ and $y = \beta _t^{n} - \hat{\beta }_{t \pm 1}$, we deduce that for all $\beta _t \in \mathbb {R}^p$,

$$\begin{aligned} \Vert \beta _t - \hat{\beta }_{t - 1}\Vert _2&\le \Vert \beta _t^{n} - \hat{\beta }_{t - 1}\Vert _2 + \frac{ ( \beta _t - \hat{\beta }_{t - 1})'(\beta _t - \beta _t^n) }{\Vert \beta _t^n - \hat{\beta }_{t - 1} \Vert _2 } + \frac{\Vert \beta _t - \beta _t^n \Vert _2^2 }{2\Vert \beta _t^n - \hat{\beta }_{t - 1} \Vert _2} , \end{aligned}$$

(44)

$$\begin{aligned} \Vert \beta _t - \hat{\beta }_{t + 1}\Vert _2&\le \Vert \beta _t^{n} - \hat{\beta }_{t + 1}\Vert _2 + \frac{ ( \beta _t - \hat{\beta }_{t + 1})'(\beta _t - \beta _t^n) }{\Vert \beta _t^n - \hat{\beta }_{t +1} \Vert _2 } + \frac{\Vert \beta _t - \beta _t^n \Vert _2^2 }{ 2 \Vert \beta _t^n - \hat{\beta }_{t + 1} \Vert _2} . \end{aligned}$$

(45)

In addition, it is immediate that

$$\begin{aligned} \Vert \beta _t - z_t \Vert _2^2&= \Vert \beta _t^n - z_t \Vert _2^2 - 2 (\beta _n - z_t)'(\beta _t - \beta _t^n ) + \Vert \beta _t - \beta _t^n \Vert _2^2 . \end{aligned}$$

(46)

Multiplying (44) by $\lambda _2 w_{t-1}$, (45) by $\lambda _2 w_{t}$, (46) by $L_t/2$, summing these relations, and adding $g_1(\beta _t)$ on each side, we obtain

$$\begin{aligned} (g_1 + g_2)(\beta _t) \le g_1(\beta _t) + g_2(\beta _t^n) + \nabla g_2(\beta _t^n) ' (\beta _t - \beta _t^n) + \frac{1}{2\gamma _n} \Vert \beta _t - \beta _t^n \Vert _2^2 . \end{aligned}$$

(47)

The left-hand side of (47) is simply $\bar{g}(\beta _t)$. Also, in view of Remark 2, the minimizer of the right-hand side of (47) is $\mathcal {T}(\beta _t^n) = \beta _t^{n+1}$. Evaluating (47) at $\beta _t^{n+1}$ and exploiting this minimizing property, it follows that

$$\begin{aligned} \begin{aligned} \bar{g}(\beta _t^{n+1} )&\le g_1(\beta _t^{n+1} ) + g_2(\beta _t^n) + \nabla g_2(\beta _t^n) ' (\beta _t^{n+1} - \beta _t^n) + \frac{1}{2\gamma _n} \Vert \beta _t^{n+1} - \beta _t^n \Vert _2^2 \\&\le g_1(\beta _t^{n} ) + g_2(\beta _t^n) + \nabla g_2(\beta _t^n) ' (\beta _t^n - \beta _t^n) + \frac{1}{2\gamma _n} \Vert \beta _t^{n} - \beta _t^n \Vert _2^2 \\&= \bar{g}(\beta _t^n). \end{aligned} \end{aligned}$$

(48)

This establishes the desired descent property.

Prerequisites of Theorem 2 of Bredies and Lorenz (2008). The distance $r_n = \bar{g}(\beta _t^n) - \bar{g}(\beta _t^*)$ to the minimum of the objective can be usefully decomposed as

$$\begin{aligned} \begin{aligned} r_n&= R(\beta _t^n) + T(\beta _t^n) \\ R(\beta _t)&= \nabla g_2(\beta _t^*) '( \beta _t - \beta _t^*) + g_1(\beta _t) - g_1(\beta _t^*) \\ T(\beta _t)&= g_2(\beta _t) - g_2(\beta _t^*) - \nabla g_2(\beta _t^*)'( \beta _t - \beta _t^*) \end{aligned} \end{aligned}$$

(49)

where $R(\beta _t)$ is a Bregman-like distance and $T(\beta _t)$ is the remainder of the Taylor expansion of $g_2$ at $\beta _t^*$.

To obtain the linear convergence of $(\beta _t^n)$ to $\beta _t^*$ and the exponential decay of $(r_n)$ to 0 with Theorem 2 of Bredies and Lorenz (2008), it suffices to show that

$$\begin{aligned} \Vert \beta _t - \beta _t^*\Vert _2^2 \le c \left( R(\beta _t) + T(\beta _t)\right) \end{aligned}$$

(50)

for some constant $c>0$ and for all $\beta _t\in \mathbb {R}^p$.

Invoking the convexity of $\Vert \cdot \Vert _2$ and strong convexity of $\Vert \cdot \Vert _2^2$, one sees that $T(\beta _t) \ge (L_t/2) \Vert \beta _t - \beta _t^*\Vert _2^2$ for all $\beta _t$. Also, $R(\beta _t) \ge 0$ for all $\beta _t$ (Lemma 2 of Bredies and Lorenz 2008) so that c can be taken as $2/L_t$ in (50). $\square$

Proof of Theorem 2

We first observe that by design, each of the four steps or components of Algorithm 5 is nonincreasing in the objective function F. Indeed the first three steps (optimization with respect to single blocks, single chains, and descent over fixed chains) are all based on FISTA (Algorithms 3 and 4) which is globally convergent (Beck and Teboulle 2009, Theorem 4.4). As each of these components minimizes F under certain constraints (namely, some blocks or fusion chains are fixed), the objective value of their output, say $F(\beta ^{n+1})$, cannot be lower than that of their input, $F(\beta ^n)$. The fourth step, subgradient descent is also nonincreasing because the subgradient of minimum norm—if it is not zero—provides a direction of (steepest) descent. The line search for the step size in the subgradient step then guarantees that the objective does not decrease after this step. As a result, Algorithm 5 as a whole is nonincreasing in F.

Let us denote a generic segmentation of the set $\{ 1, \ldots , T\}$ by $C=(C_1,\ldots , C_K)$ where $C_k =\{ T_k ,\ldots , T_{k+1} - 1\}$ and $1= T_1 \le \cdots \le T_K < T_{K+1}=T+1$. There are $2^{T-1}$ such segmentations. Let $S_C$ be the associated open set for the parameter $\beta$:

$$\begin{aligned} S_C = \left\{ \beta \in \mathbb {R}^{pT}: \beta _{T_k} = \cdots = \beta _{T_{k+1}-1},\ \beta _{T_{k+1}-1} \ne \beta _{T_{k+1}} , 1\le k \le K \right\} . \end{aligned}$$

To each segmentation C is associated an infimum value of F: $\inf _{\beta \in S_C}F(\beta )$. Let $(\beta ^n)_{n\ge 0}$ be the sequence of iterates generated by Algorithm 5 and let $C^{n}$ the associated segmentations of $\{1,\ldots ,T\}$. By setting the tolerance $\epsilon$ of Algorithm 5 sufficiently small, each time the third optimization component (descent over fixed chains) is applied, say with $\beta ^n$ as input and $\beta ^{n+1}$ as output, the objective $F(\beta )$ can be made arbitrarily close to $\min _{ \beta \in S_{C^n}}F(\beta )$ or even become inferior to this value if a fusion of chains occurs during this optimization. If the segmentation $C^n$ is optimal, i.e. $\min _{ S_{C^n} } F (\beta ) = \min _{\beta \in \mathbb {R}^{pT}}F (\beta )$, then Algorithm 5 has converged: for all subsequent iterates $m\ge n$, $F(\beta ^m)$ and $\beta ^m$ will stay arbitrarily close to the minimum of F and to the set of minimizers, respectively, because of the nonincreasing property of Algorithm 5. If the segmentation $C^n$ is suboptimal, i.e. $\min _{\beta \in S_{C^n} } F(\beta ) > \min _{\beta \in \mathbb {R}^{pT}}F (\beta )$, provided that $\epsilon$ is sufficiently small, the (fourth) subgradient step of Algorithm 5 will eventually produce an iterate $\beta ^m$ ($m \ge n$) such that $F(\beta ^m) < \min _{ \beta \in S_{C^n} }F(\beta )$. This is because each subgradient step brings the iterates closer to the set of global minimizers of F. Once this has happened, the nonincreasing property of the algorithm guarantees that the segmentation $C^n$ will not be visited again. Because the segmentations of $\{ 1,\ldots ,T\}$ are in finite number, Algorithm 5 eventually finds an optimal segmentation C such that $\min _{\beta \in S_C} F(\beta ) = \min _{\beta \in \mathbb {R}^{pT}} F(\beta )$. Then, through its third level of optimization (descent over fixed chains), it reaches the global minimum of F. We note that the first and second components of Algorithm 5 (block coordinate descent over single blocks and single chains) are not necessary to ensure global convergence; they only serve for computational speed. $\square$