Elsevier

Operations Research Letters

Volume 45, Issue 6, November 2017, Pages 565-569
Operations Research Letters

Descent line search scheme using Gers̆gorin circle theorem

https://doi.org/10.1016/j.orl.2017.08.010Get rights and content

Abstract

In the context of the minimization of a real function, we propose a line search scheme that involves a new positive definite modification of the Hessian. In this framework, a safeguard based on Gers̆gorin Circle’s theorem provides an approximation of the Hessian that improves with iteration count. Convergence analysis of the scheme is validated by numerical experiments.

Introduction

Line search method is one of the most competent, effectual and expeditious methods for solving optimization problem, that has been a significant area of fascination and concern for the researchers (see [3], [9], [10], [12], [13]). The general structure of a descent line search iterative scheme for a function f:RnR is x(k+1)=x(k)+αkp(k), where p(k) is a descent direction of f at x(k) and αk is the step length at x(k) along p(k). For Newton’s method, p(k) is 2f(x(k))1f(x(k)), where 2f(x(k)) and f(x(k)) represent the Hessian and gradient of f at x(k) respectively. In this method if the initial guess is chosen far from the solution point, then the Hessian of the objective function may not be positive definite at every iterating point. In such case, the Hessian matrix can be modified to an approximate positive definite matrix to ensure the descent property of the scheme. Several techniques on positive definite modification of Hessian matrix are summarized in Section 3.4, Chapter 3, [7], which are based on computing eigenvalues using two basic strategies. One of these strategies directly computes the eigenvalues of the Hessian matrix at the current iteration point and then suitably adds a diagonal matrix of the form τI, where τ=max(0,ηλmin(2f)), η and λmin(2f) being a small positive number and the minimum eigenvalue of the Hessian of f respectively. This is a computationally expensive process in higher dimension. Another strategy is based on the concept of modified symmetric indefinite factorization of the Hessian matrix (see [4], [8]). This method factorizes the permuted Hessian matrix into LBLT with a lower triangular matrix L and a block diagonal matrix B of at most 2 × 2 blocks, which makes the process computationally easier for computing eigenvalues in comparison to the first strategy.

In this paper, we propose a new approach that does not compute eigenvalues, not even implicitly, at any stage like earlier studies. This process first allows a positive definite safeguard at every iteration, and then backtracks sequentially to the Hessian matrix using Gers̆gorin circle theorem. The possibility of Cholesky factorization is verified once in each iteration for investigating the positive definiteness of a matrix. A real positive sequence (converging to 1), is assigned during the iteration process for generating the backtracking step. The global convergence property of the proposed scheme is established with Wolfe inexact line search under Zoutendijk condition. Further, it is proved that the modified matrix at each iteration converges to the Hessian matrix at the solution point. This fact ensures the superlinear convergence property of the proposed scheme. The computational experience on a set of test problems is provided for numerical support. The performance profiles for the number of iterations, the number of function evaluations, and the elapsed execution time on this test set are also presented.

The following two existing results are required to proceed for the theoretical development of this paper. The first result is based on Gers̆gorin Circle Theorem [6], [11].

Theorem 1.1

Let A be a complex matrix of order n, with entries aij. For j{1,2,3,,n}, let Ri=ji|aij| and D(aii,Ri) be the closed disc centred at aii with radius Ri. Such a disc is known as Ger s̆gorin disc. Every eigenvalue λ of A lies within at least one of the Ger s̆gorin disc D(aii,Ri), that is, λaiiRi for some i.

Theorem 1.2 Zoutendijk Theorem [14], Theorem No. 3.2 of [7]

Consider kth iteration of an optimization algorithm for minimizing f(x),xRn in the form x(k+1)=x(k)+αkp(k), where p(k) is a descent direction and αk satisfies Wolfe condition. Suppose f is bounded below in Rn and continuously differentiable in an open set containing the level set ={x:f(x)f(x(0))}, where x(0) is the starting point of the iteration. Assume also that f is Lipschitz continuous on . That is, there exists a constant L>0 such that f(x)f(x̃)<Lxx̃x,x̃. Then k0cos2θkf(x(k))2<, where θk is the angle between p(k) and f(x(k)).

The rest of the paper is organized as follows. Section 2 proposes the idea of the new scheme and Section 3 describes the algorithm and convergence of the scheme. A detailed computational experience is illustrated in Section 4 and some concluding remarks are provided in Section 5.

Section snippets

Proposing new line search scheme

Consider an optimization problem (P):minxRnf(x),where f is twice differentiable. We propose a Newton-like scheme that modifies kth iteration point as x(k+1)=x(k)+αkp(k), where αk is the step length and p(k) is the descent direction to move along. Generally, p(k) is chosen as D(k)1f(x(k)), where D(k) is a positive definite approximation of the Hessian matrix. In this framework, D(k) is formed based on Gers̆gorin Circle theorem which is used as a positive definite safeguard to the Hessian,

Convergence of the scheme

Consider f to be twice continuously differentiable. First, we recall Theorem 3.6 from [7], which shows that if the search direction approximates the Newton-direction well enough, then the unit step length will satisfy the Wolfe conditions as the iterates converge to the solution. Now, in addition to the assumptions of Theorem 3.6 from [7], we consider that the condition number κ(D(k)) is uniformly bounded for each k. Then from the discussion of Section 3.2 of [7], global convergence of the

Numerical illustration

For numerical illustration, we run the proposed algorithm on a test set in MATLAB-R2014a platform with 8 GB RAM. The test set is formed by taking 25 functions of 1000 dimension and 5 functions of 500 dimension from [1]. In the construction of the proposed scheme, one should choose a monotonically convergent sequence {ck}1 with 0<c0<1. Here this sequence is fixed as ck+1=1ckk5. However, the choice of different {ck} may be made since the results of this scheme remain still preserved. The

Conclusion

In this paper, a line search method for unconstrained optimization problem with a new approach is proposed by modifying the Hessian matrix. A suitable descent sequence is developed using a backtracking process and the advantage of the proposed scheme over the other two existing methods is justified through numerical tests and their respective performance profiles. This backtracking Hessian modification approach can be used to other line search techniques of numerical optimization, which is kept

Acknowledgements

The authors thank the editor and reviewers for their constructive comments to improve the quality and clarity of the paper.

References (14)

There are more references available in the full text version of this article.

Cited by (0)

View full text