A Derivative-Free Nonlinear Least Squares Solver

Abstract

An improved version of derivative-free nonlinear least squares iterative solver developed earlier by the author is described. First, we apply a regularization technique to stabilize the evaluation of search directions similar to the one used in the Levenberg-Marquardt methods. Second, we propose several modified designs for the rectangular preconditioning matrix, in particular a sparse adaptive techniques avoiding the use of pseudorandom sequences. The resulting algorithm is based on easily parallelizable computational kernels such as dense matrix factorizations and elementary vector operations thus having a potential for an efficient implementation on modern high-performance computers. Numerical results are presented for several standard test problems as well as for some special complex-valued cases to demonstrate the effectiveness of the proposed improvements to method.

A Limiting Stepsize Along Subnormalized Direction

Similar to [6, 9], the proof of (9) is based on the assumption that the limiting stepsize $\widehat{\alpha }=\widehat{\alpha }(f,p)$ along a subnormalized direction p exists such that the limiting stepsize condition

$$\begin{aligned} \Vert f(x+\alpha p)-f-\alpha Jp\Vert \le \left( \alpha -\frac{\alpha ^2}{2}\right) \frac{\Vert Jp\Vert ^2}{\Vert f\Vert } \end{aligned}$$

(26)

is satisfied for all $0<\alpha \le \widehat{\alpha }$. (To clarify the notations, further we will omit the iteration index t.) For instance, a sufficient condition for (26) to hold is that f satisfies the local Lipschitz condition at x and J(x) has full column rank, see, e.g., [9]. Indeed, (9) can be obtained from (26) and (8) as follows:

$$ \Vert f(x+\alpha p)\Vert \le \Vert f+\alpha Jp\Vert + \Vert f(x+\alpha p)-f-\alpha Jp\Vert $$

$$ = \left( \Vert f\Vert ^2+2\alpha f^{\top }Jp+\alpha ^2\Vert Jp\Vert ^2\right) ^{1/2} + \Vert f(x+\alpha p)-f-\alpha Jp\Vert $$

$$ \le \left( \Vert f\Vert ^2 - 2\alpha \Vert Jp\Vert ^2 + \alpha ^2\Vert Jp\Vert ^2\right) ^{1/2} + \left( \alpha -\frac{\alpha ^2}{2}\right) \frac{\Vert Jp\Vert ^2}{\Vert f\Vert } $$

$$ = \Vert f\Vert \left( \left( 1 - (2\alpha -\alpha ^2)\frac{\Vert Jp\Vert ^2}{\Vert f\Vert ^2}\right) ^{1/2} + \left( \alpha -\frac{\alpha ^2}{2}\right) \frac{\Vert Jp\Vert ^2}{\Vert f\Vert ^2}\right) $$

$$ \le \Vert f\Vert \left( 1 - \left( \left( \alpha -\frac{\alpha ^2}{2}\right) \frac{\Vert Jp\Vert ^2}{\Vert f\Vert ^2}\right) ^2\right) ^{1/2}, $$

where the latter estimate follows from the inequality

$$ \sqrt{1-\eta } + \frac{\eta }{2} \le \sqrt{1-\frac{\eta ^2}{4}}, $$

which holds for any $0\le \eta \le 1$ and is used with $\eta =\alpha (2-\alpha )\Vert Jp\Vert ^2/\Vert f\Vert ^2$, see also (11).