A method to accelerate the rate of convergence of a class of optimization algorithms

Abstract

Supposez ∈ Eⁿ is a solution to the optimization problem minimizeF(x) s.t.x ∈ Eⁿ and an algorithm is available which iteratively constructs a sequence of search directions {s _j } and points {x _j } with the property thatx _j →z. A method is presented to accelerate the rate of convergence of {x _j } toz provided that n consecutive search directions are linearly independent. The accelerating method uses n iterations of the underlying optimization algorithm. This is followed by a special step and then another n iterations of the underlying algorithm followed by a second special step. This pattern is then repeated. It is shown that a superlinear rate of convergence applies to the points determined by the special step. The special step which uses only first derivative information consists of the computation of a search direction and a step size. After a certain number of iterations a step size of one will always be used. The acceleration method is applied to the projection method of conjugate directions and the resulting algorithm is shown to have an (n + 1)-step cubic rate of convergence. The acceleration method is based on the work of Best and Ritter [2].