The Dai–Liao nonlinear conjugate gradient method with optimal parameter choices

Highlights

• Solutions for an open problem in the conjugate gradient methods are discussed.

• A singular value study on the Dai–Liao conjugate gradient method is made.

• Two modified Dai–Liao conjugate gradient methods are suggested.

• Convergence analyses and numerical comparisons are made.

Abstract

Minimizing two different upper bounds of the matrix which generates search directions of the nonlinear conjugate gradient method proposed by Dai and Liao, two modified conjugate gradient methods are proposed. Under proper conditions, it is briefly shown that the methods are globally convergent when the line search fulfills the strong Wolfe conditions. Numerical comparisons between the implementations of the proposed methods and the conjugate gradient methods proposed by Hager and Zhang, and Dai and Kou, are made on a set of unconstrained optimization test problems of the CUTEr collection. The results show the efficiency of the proposed methods in the sense of the performance profile introduced by Dolan and Moré.

Introduction

Conjugate gradient (CG) methods comprise a class of unconstrained optimization algorithms characterized by low memory requirements and strong global convergence properties (Dai et al., 1999) which made them popular for engineers and mathematicians engaged in solving large-scale problems in the following form:

$${\displaystyle \underset{x\in {\mathbb{R}}^n}{\min }}f(x)\text{,}$$

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is a smooth nonlinear function and its gradient is available. The iterative formula of a CG method is given by

$$x_0\in {\mathbb{R}}^n\text{,}{x}_{k+1}=x_k+s_k\text{,}{s}_k={\alpha }_k{d}_k\text{,}k=0\text{,}1\text{,}\dots \text{,}$$

in which ${\alpha }_k$ is a steplength to be computed by a line search procedure and $d_k$ is the search direction defined by

$$d_0=-g_0\text{,}{d}_{k+1}=-g_{k+1}+{\beta }_k{d}_k\text{,}k=0\text{,}1\text{,}\dots \text{,}$$

where $g_k=\nabla f(x_k)$ and ${\beta }_k$ is a scalar called the CG (update) parameter.

The steplength ${\alpha }_k$ is usually chosen to satisfy certain line search conditions (Sun & Yuan, 2006). Among them, the so-called strong Wolfe conditions (Wolfe, 1969) have attracted special attention in the convergence analyses and the implementations of CG methods, requiring that

$$f(x_k+{\alpha }_k{d}_k)-f(x_k)⩽\delta {\alpha }_k\nabla f{(x_k)}^T{d}_k\text{,}$$

$$|\nabla f{(x_k+{\alpha }_k{d}_k)}^T{d}_k|⩽-\sigma \nabla f{(x_k)}^T{d}_k\text{,}$$

where $0<\delta <\sigma <1$.

Different choices for the CG parameter lead to different CG methods (Hager & Zhang, 2006b). Based on an extended conjugacy condition, one of the essential CG methods has been proposed by Dai and Liao (2001) (DL), with the following CG parameter:

$${\beta }_k^{\mathit{DL}}=\frac{g_{k+1}^T{y}_k}{d_k^T{y}_k}-t\frac{g_{k+1}^T{s}_k}{d_k^T{y}_k}\text{,}$$

where t is a nonnegative parameter and $y_k=g_{k+1}-g_k$. Note that if $t=0$, then ${\beta }_k^{\mathit{DL}}$ reduces to the CG parameter proposed by Hestenes and Stiefel (1952). Also, the CG parameter proposed by Hager and Zhang (2005) (HZ), i.e.

$${\beta }_k^{\mathit{HZ}}=\frac{g_{k+1}^T{y}_k}{d_k^T{y}_k}-2\frac{\Vert y_k{\Vert }^2}{d_k^T{y}_k}\frac{g_{k+1}^T{d}_k}{d_k^T{y}_k}\text{,}$$

can be viewed as an adaptive version of (1.5) corresponding to $t=2\frac{\Vert y_k{\Vert }^2}{s_k^T{y}_k}$, where $\Vert ·\Vert$ denotes the Euclidean norm. Similarly, the CG parameter suggested by Dai and Kou (2013) (DK), i.e.

$${\beta }_k({\tau }_k)=\frac{g_{k+1}^T{y}_k}{d_k^T{y}_k}-\left({\tau }_k+\frac{\Vert y_k{\Vert }^2}{s_k^T{y}_k}-\frac{s_k^T{y}_k}{\Vert s_k{\Vert }^2}\right)\frac{g_{k+1}^T{s}_k}{d_k^T{y}_k}\text{,}$$

in which ${\tau }_k$ is a parameter corresponding to the scaling factor in the scaled memoryless BFGS method (Sun & Yuan, 2006), can be considered as another adaptive version of (1.5) corresponding to $t={\tau }_k+\frac{\Vert y_k{\Vert }^2}{s_k^T{y}_k}-\frac{s_k^T{y}_k}{\Vert s_k{\Vert }^2}$. In (Dai & Liao, 2001), it has been shown that a CG method in the form of (1.1), (1.2) with ${\beta }_k={\beta }_k^{\mathit{DL}}$ is globally convergent for uniformly convex functions.

The approach of Dai and Liao has been paid special attention to by many researches. In several efforts, modified secant equations have been applied to make modifications on the DL method. For example, Yabe and Takano (2004) used the modified secant equation proposed by Zhang, Deng, and Chen (1999). Also, Zhou and Zhang (2006) applied the modified secant equation proposed by Li and Fukushima (2001). Li, Tang, and Wei (2007) used the modified secant equation proposed by Wei, Li, and Qi (2006). Ford, Narushima, and Yabe (2008) employed the multi-step quasi-Newton equations proposed by Ford and Moghrabi (1994). Babaie-Kafaki, Ghanbari, and Mahdavi-Amiri (2010) applied a revised form of the modified secant equation proposed by Zhang et al. (1999) and the modified secant equation proposed by Yuan (1991). Furthermore, in several other attempts, the modified versions of ${\beta }_k^{\mathit{DL}}$ suggested in (Babaie-Kafaki et al., 2010, Ford et al., 2008, Li et al., 2007, Yabe and Takano, 2004, Zhou and Zhang, 2006) have been used to achieve descent CG methods. Examples include the studies made by Narushima and Yabe, 2012, Sugiki et al., 2012, Livieris and Pintelas, 2012.

Here, based on a singular value study on the DL method, two nonlinear CG methods are proposed. The remainder of this work is organized as follows. In Section 2, the methods are suggested and their global convergence analysis is discussed. In Section 3, they are numerically compared with the CG methods proposed by Hager and Zhang, and Dai and Kou, and comparative testing results are reported. Finally, conclusions are made in Section 4.