Reminiscences on the development of the variational approach to Davidon’s variable-metric method

Abstract.

In the mid-1960’s, Davidon’s method was brought to the author’s attention by M.J.D. Powell, one of its earliest proponents. Its great efficacy in solving a rather difficult computational problem in which the author was involved led to an attempt to find a “best” updating formula. “Best” seemed to suggest “least” in the sense of some norm, to further the stability of the method. This led to the idea of minimizing a generalized quadratic (Frobenius) norm with the quasi-Newton and symmetry constraints on the updates. Several interesting formulas were derived, including the Davidon-Fletcher-Powell formula (as shown by Goldfarb). This approach was extended to the derivation of updates requiring no derivatives, and to Broyden-like updates for the solution of simultaneous nonlinear equations. Attempts were made to derive minimum-norm corrections in product-form updates, with an eye to preserving positive-definiteness. In the course of this attempt, it was discovered that the DFP formula could be written as a product, leading to some interesting theoretical developments. Finally, a linearized product-form update was developed which was competitive with the best update (BFGS) of that time.