Convergence of Liu–Storey conjugate gradient method

Abstract

The conjugate gradient method is a useful and powerful approach for solving large-scale minimization problems. Liu and Storey developed a conjugate gradient method, which has good numerical performance but no global convergence result under traditional line searches such as Armijo, Wolfe and Goldstein line searches. In this paper a convergent version of Liu–Storey conjugate gradient method (LS in short) is proposed for minimizing functions that have Lipschitz continuous partial derivatives. By estimating the Lipschitz constant of the derivative of objective functions, we can find an adequate step size at each iteration so as to guarantee the global convergence and improve the efficiency of LS method in practical computation.

Introduction

The conjugate gradient method is an approach for solving large scale minimization problems due to its smaller storage requirements and simple computation (e.g. [3], [5], [9], [10], [22], [23]). This method was motivated by Hestence and Stiefel in solving symmetric positive definite linear equations [16] and developed by Fletcher and Reeves (e.g. [6], [7], [10]) in solving unconstrained minimization problems. The conjugate gradient methods have wide applications in many fields, such as control science, engineering, management science and operations research, etc. [9], [29].

Consider an unconstrained minimization problem

$$\min f(x)\text{,}x\in R^n\text{,}$$

where Rⁿ denotes an n-dimensional Euclidean space and f : Rⁿ → R¹ is a continuously differentiable function.

Many approaches to solving (1) are iterative methods, such as line search and trust region methods. For sake of convenience, if x_k is the current iterate then we denote f(x_k) by f_k, ∇f(x_k) by g_k, ∇²f(x_k) by G_k, and f(x^∗) by f^∗, respectively. If G_k is available and invertible then $d_k=-G_k^{-1}{g}_k$ leads to the Newton method and d_k = −g_k results in the steepest descent method (e.g. [10], [16], [19]). The conjugate gradient method has the form

$$x_{k+1}=x_k+{\alpha }_k{d}_k\text{,}k=0\text{,}1\text{,}2\text{,}\dots \text{,}$$

where α_k is a step size and d_k is a search direction of f(x) at the current iterate x_k that takes the form

$$d_k=\left\{\begin{array}{c}-g_k\text{,}\text{if}k=0\text{;}\hfill \\ -g_k+{\beta }_k{d}_{k-1}\text{,}\text{if}k⩾1\text{,}\hfill \end{array}\right.$$

where β_k can be defined by

$$\begin{array}{cc}\hfill & {\beta }_k^{\text{FR}}=\frac{\Vert g_k{\Vert }^2}{\Vert g_{k-1}{\Vert }^2}\text{,}{\beta }_k^{\text{PRP}}=\frac{g_k^T(g_k-g_{k-1})}{\Vert g_{k-1}{\Vert }^2}\text{,}\hfill \\ \hfill & {\beta }_k^{\text{HS}}=\frac{g_k^T(g_k-g_{k-1})}{d_{k-1}^T(g_k-g_{k-1})}\text{,}{\beta }_k^{\text{LS}}=-\frac{g_k^T(g_k-g_{k-1})}{d_{k-1}^T{g}_{k-1}}\text{,}\hfill \\ \hfill & {\beta }_k^{\text{CD}}=-\frac{g_k^T{g}_k}{d_{k-1}^T{g}_{k-1}}\text{,}{\beta }_k^{\text{DY}}=\frac{g_k^T{g}_k}{(g_k-g_{k-1}{)}^T{d}_{k-1}}\hfill \end{array}$$

or by other formulae (e.g. see [9], [24]). The corresponding method is respectively called FR (Fletcher–Revees [10]), PRP (Polak–Ribiére–Polyak [20], [21]), HS (Hestenes–Stiefel [16]), LS (Liu–Storey [18]), CD (conjugate descent [9]) and DY (Dai–Yuan [7]) conjugate gradient method.

Although the above mentioned conjugate gradient algorithms are equivalent to each other for minimizing strong convex quadratic functions under exact line search, they have different performance when using them to minimize non-quadratic functions or using inexact line searches.

For non-quadratic objective functions, the FR method has global convergence when exact line search or strong Wolfe line search [2], [6] is used. The PRP method has no global convergence under some traditional line searches. Some convergent versions were proposed by using some new complicated line searches or through restricting the parameter β_k to a non-negative number [13], [14], [15], [25], [26]. The CD method was proved to have global convergence property under strong Wolfe line search with a strong restriction on the parameters [5] and DY method has global convergence under weak Wolfe line search [7]. Some impressive literature on conjugate gradient methods can be found in [3], [4], [5], [8], [11], [12], [18], [17], [22], [23], [30].

However, to the best of our knowledge, the global convergence of the original LS and HS methods has not been proved under all the mentioned line searches. In this paper, we propose a new line search procedure and investigate the global convergence of the original LS method.

In the line search method, the search direction d_k is generally required to satisfy

$$g_k^T{d}_k<0\text{,}$$

which guarantees that d_k is a descent direction of f(x) at x_k [10], [16], [29]. In order to guarantee the global convergence, we sometimes require d_k to satisfy the sufficient descent condition

$$g_k^T{d}_k⩽-c\Vert g_k{\Vert }^2\text{,}$$

where c > 0 is a constant. Moreover, the angle property is often used in proving the global convergence of related line search methods, that is

$$\cos 〈-g_k\text{,}{d}_k〉=-\frac{g_k^T{d}_k}{\Vert g_k\Vert ·\Vert d_k\Vert }⩾\tau \text{,}$$

where 1 ⩾ τ > 0.

Once the descent direction d_k is determined at the kth iteration we should seek a step size along the descent direction and complete one iteration.

There are many approaches to find an available step size in which the exact line search is difficult or even impossible to carry out and some inexact line search rules are sometimes useful and powerful in practical computation, such as Armijo line search [1], Goldstein and Wolfe line searches [9], [19], [29]. The Armijo line search is useful and easy to implement in practical computation.

Armijo line search. Let s > 0 be a constant, ρ ∈ (0, 1) and μ ∈ (0, 1). Choose α_k to be the largest α in {s, sρ, sρ², … , } such that

$$f_k-f(x_k+\alpha d_k)⩾-\alpha \mu g_k^T{d}_k\text{.}$$

The drawback of the Armijo line search is how to choose the initial step size s. If s is too large then the procedure needs to call much more function evaluations. If s is too small then the efficiency of related algorithm will be decreased. Thereby, we should choose an adequate initial step size s at each iteration so as to find the step size α_k easily.

In this paper we propose a new Armijo-type line search in which an appropriate initial step size s is defined and varies at each iteration. The new Armijo-type line search enables us to find the step size α_k easily at each iteration and guarantees the global convergence of the original LS conjugate gradient method under some mild conditions. The global convergence and linear convergence rate are analyzed and numerical results show that LS method with the new Armijo-type line search is more effective than other similar methods in solving large scale minimization problems.

The rest of this paper is organized as follows. In the next section we introduce a new Armijo-type line search and establish a convergent version of LS method. In Sections 3 Global convergence, 4 Linear convergence rate the global convergence and linear convergence rate are analyzed respectively. Some numerical results are reported in Section 5.