Descent line search scheme using Gers̆gorin circle theorem
Introduction
Line search method is one of the most competent, effectual and expeditious methods for solving optimization problem, that has been a significant area of fascination and concern for the researchers (see [3], [9], [10], [12], [13]). The general structure of a descent line search iterative scheme for a function is , where is a descent direction of at and is the step length at along . For Newton’s method, is , where and represent the Hessian and gradient of at respectively. In this method if the initial guess is chosen far from the solution point, then the Hessian of the objective function may not be positive definite at every iterating point. In such case, the Hessian matrix can be modified to an approximate positive definite matrix to ensure the descent property of the scheme. Several techniques on positive definite modification of Hessian matrix are summarized in Section 3.4, Chapter 3, [7], which are based on computing eigenvalues using two basic strategies. One of these strategies directly computes the eigenvalues of the Hessian matrix at the current iteration point and then suitably adds a diagonal matrix of the form , where and being a small positive number and the minimum eigenvalue of the Hessian of respectively. This is a computationally expensive process in higher dimension. Another strategy is based on the concept of modified symmetric indefinite factorization of the Hessian matrix (see [4], [8]). This method factorizes the permuted Hessian matrix into with a lower triangular matrix and a block diagonal matrix of at most 2 2 blocks, which makes the process computationally easier for computing eigenvalues in comparison to the first strategy.
In this paper, we propose a new approach that does not compute eigenvalues, not even implicitly, at any stage like earlier studies. This process first allows a positive definite safeguard at every iteration, and then backtracks sequentially to the Hessian matrix using Gergorin circle theorem. The possibility of Cholesky factorization is verified once in each iteration for investigating the positive definiteness of a matrix. A real positive sequence (converging to 1), is assigned during the iteration process for generating the backtracking step. The global convergence property of the proposed scheme is established with Wolfe inexact line search under Zoutendijk condition. Further, it is proved that the modified matrix at each iteration converges to the Hessian matrix at the solution point. This fact ensures the superlinear convergence property of the proposed scheme. The computational experience on a set of test problems is provided for numerical support. The performance profiles for the number of iterations, the number of function evaluations, and the elapsed execution time on this test set are also presented.
The following two existing results are required to proceed for the theoretical development of this paper. The first result is based on Gergorin Circle Theorem [6], [11].
Theorem 1.1 Let
be a complex matrix of order n, with entries
. For
, let
and
be the closed disc centred at
with radius
. Such a disc is known as Ger
gorin disc. Every eigenvalue
of
lies within at least one of the Ger
gorin disc
, that is,
for some
.
Theorem 1.2 Zoutendijk Theorem [14], Theorem No. 3.2 of [7] Consider
th iteration of an optimization algorithm for minimizing
in the form
, where
is a descent direction and
satisfies Wolfe condition. Suppose
is bounded below in
and continuously differentiable in an open set containing the level set
, where
is the starting point of the iteration. Assume also that
is Lipschitz continuous on
. That is, there exists a constant
such that
. Then
where
is the angle between
and
.
The rest of the paper is organized as follows. Section 2 proposes the idea of the new scheme and Section 3 describes the algorithm and convergence of the scheme. A detailed computational experience is illustrated in Section 4 and some concluding remarks are provided in Section 5.
Section snippets
Proposing new line search scheme
Consider an optimization problem where is twice differentiable. We propose a Newton-like scheme that modifies th iteration point as , where is the step length and is the descent direction to move along. Generally, is chosen as , where is a positive definite approximation of the Hessian matrix. In this framework, is formed based on Gergorin Circle theorem which is used as a positive definite safeguard to the Hessian,
Convergence of the scheme
Consider to be twice continuously differentiable. First, we recall Theorem 3.6 from [7], which shows that if the search direction approximates the Newton-direction well enough, then the unit step length will satisfy the Wolfe conditions as the iterates converge to the solution. Now, in addition to the assumptions of Theorem 3.6 from [7], we consider that the condition number is uniformly bounded for each . Then from the discussion of Section 3.2 of [7], global convergence of the
Numerical illustration
For numerical illustration, we run the proposed algorithm on a test set in MATLAB-R2014a platform with 8 GB RAM. The test set is formed by taking 25 functions of 1000 dimension and 5 functions of 500 dimension from [1]. In the construction of the proposed scheme, one should choose a monotonically convergent sequence with . Here this sequence is fixed as . However, the choice of different may be made since the results of this scheme remain still preserved. The
Conclusion
In this paper, a line search method for unconstrained optimization problem with a new approach is proposed by modifying the Hessian matrix. A suitable descent sequence is developed using a backtracking process and the advantage of the proposed scheme over the other two existing methods is justified through numerical tests and their respective performance profiles. This backtracking Hessian modification approach can be used to other line search techniques of numerical optimization, which is kept
Acknowledgements
The authors thank the editor and reviewers for their constructive comments to improve the quality and clarity of the paper.
References (14)
- et al.
New spectral PRP conjugate gradient method for unconstrained optimization
Appl. Math. Lett.
(2011) On the convergence of the modified Levenberg–Marquardt method with a nonmonotone second order Armijo type line search
J. Comput. Appl. Math.
(2013)An unconstrained optimization test functions collection
Adv. Model. Optim
(2008)- et al.
Some stable methods for calculating inertia and solving symmetric linear systems
Math. Comp.
(1977) - et al.
A dwindling filter line search method for unconstrained optimization
Math. Comp.
(2015) - et al.
A modified Cholesky algorithm based on a symmetric indefinite factorization
SIAM J. Matrix Anal. Appl.
(1998) - et al.
Benchmarking optimization software with performance profiles
Math. Program.
(2002)