On an Elliptical Trust-Region Procedure for Ill-Posed Nonlinear Least-Squares Problems

Abstract

In this paper, we address the stable numerical solution of ill-posed nonlinear least-squares problems with small residual. We propose an elliptical trust-region reformulation of a Levenberg–Marquardt procedure. Thanks to an appropriate choice of the trust-region radius, the proposed procedure guarantees an automatic choice of the free regularization parameters that, together with a suitable stopping criterion, ensures regularizing properties to the method. Specifically, the proposed procedure generates a sequence that even in case of noisy data has the potential to approach a solution of the unperturbed problem. The case of constrained problems is considered, too. The effectiveness of the procedure is shown on several examples of ill-posed least-squares problems.

Appendix

1.1 Proof of item (ii) of Lemma 4.2 and item (ii) of Lemma 4.4

Proof

The proof is the same for the noise-free and the noisy case; for generality, the notation of the noisy case is employed.

Since $\lambda _k>0$ from Lemma 3.4, the trust-region is active, and from (30) it follows that

$$\begin{aligned} \Delta _k=\Vert z(\lambda _k)\Vert \le \frac{\Vert B_k^{1/2}f'_k\Vert }{\lambda _k}. \end{aligned}$$

Then, if $\Delta _k$ chosen at Step 1 of Algorithm 3.1 guarantees condition $\pi _k(p_k)\ge \eta$, the thesis follows as

$$\begin{aligned} \lambda _k \le \frac{\Vert B_k^{1/2}f'_k\Vert }{\Delta _k}\le \frac{1}{C_{\min }}=\bar{\lambda }. \end{aligned}$$

(56)

Otherwise, the trust-region radius is progressively reduced, and a bound for the value of $\Delta _k$ at termination of Step 2 of Algorithm 3.1 can be provided. First, consider the case

$$\begin{aligned} f_\delta \left( x_k^\delta +p_k\right) >\frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2. \end{aligned}$$

Trivially,

$$\begin{aligned} 1-\pi _k(p_k) = \frac{f_\delta \left( x_k^\delta +p_k\right) -\frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2}{f_\delta \left( x_k^\delta \right) -\frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2} \end{aligned}$$

and

$$\begin{aligned}&f_\delta \left( x_k^\delta +p_k\right) -\frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2 \\&\quad = \frac{1}{2}\Vert F\left( x_k^\delta +p_k\right) \pm F\left( x_k^\delta \right) \pm F'\left( x_k^\delta \right) p_k-y^\delta \Vert ^2\\&\qquad - \frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2\\&\quad = \frac{1}{2}\Vert F \left( x_k^\delta +p_k\right) -F\left( x_k^\delta \right) -F' \left( x_k^\delta \right) p_k\Vert ^2 \nonumber \\&\qquad + \Vert F\left( x_k^\delta +p_k\right) -F\left( x_k^\delta \right) -F' \left( x_k^\delta \right) p_k\Vert \\&\qquad \cdot \Vert F\left( x_k^\delta \right) -y^\delta +F'\left( x_k^\delta \right) p_k\Vert . \end{aligned}$$

By the Lipschitz continuity of $F'$ it holds

$$\begin{aligned} \Vert F\left( x_k^\delta +p_k\right) -F\left( x_k^\delta \right) -F'\left( x_k^\delta \right) p_k\Vert \le \frac{L}{2} \Vert p_k\Vert ^2. \end{aligned}$$

Moreover, using (24)

$$\begin{aligned} \Vert F\left( x_k^\delta \right) -y^\delta +F'\left( x_k^\delta \right) p(\lambda )\Vert <\Vert F\left( x_k^\delta \right) -y^\delta \Vert \end{aligned}$$

for any $\lambda \ge 0$. Consequently, as $\Vert p_k\Vert \le \Vert B_k^{1/2}\Vert \Delta _k$ and $\Delta _k\le C_{\max }\Vert B_k^{1/2} f'_k\Vert$,

$$\begin{aligned}&f_\delta \left( x_k^\delta +p_k\right) -\frac{1}{2}\Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2 \\&\quad \le \frac{L}{2} K^2 \Delta _k^2\Vert F(x_{0})-y\Vert \left( \frac{L}{4} K^6 C_{\max }^2\Vert F(x_{0})-y\Vert +1\right) . \end{aligned}$$

From [22, Theorem 6.3.1 and §8.3] it holds

$$\begin{aligned}&f_\delta \left( x_k^\delta \right) - \left( \frac{1}{2} \langle z_k, B_k^2 z_k\rangle +\langle B_k^{1/2}f'_k,z_k\rangle +f_\delta \left( x_k^\delta \right) \right) \\&\quad \ge \frac{1}{2}\Vert B_k^{1/2} f'_k\Vert \min \left\{ \Delta _k,\frac{\Vert B_k^{1/2} f'_k\Vert }{\Vert B_k^2\Vert } \right\} . \end{aligned}$$

Then, (18) yields

$$\begin{aligned} f_\delta \left( x_k^\delta \right) -\frac{1}{2} \Vert F\left( x_k^{\delta }\right) -y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2 \ge \frac{1}{2}\Delta _k \Vert B_k^{1/2}f'_k\Vert , \end{aligned}$$

whenever $\Delta _k\le \displaystyle \frac{\Vert B_k^{1/2}f'_k\Vert }{K^4}$, and this implies

$$\begin{aligned} 1-\pi _k(p_k)\le \frac{ L K^2 \Delta _k\Vert F(x_{0})-y\Vert \left( \frac{1}{4} LK^6 C_{\max }^2\Vert F(x_{0})-y\Vert +1\right) }{ \Vert B_k^{1/2}f'_k\Vert }. \end{aligned}$$

Namely, termination of the repeat loop occurs with

$$\begin{aligned} \Delta _k\le \omega \Vert B_k^{1/2} f'_k\Vert \end{aligned}$$

and

$$\begin{aligned} \omega =\min \left\{ \frac{1}{K^4},\frac{1-\eta }{ L K^2 \Vert F(x_{0})-y\Vert \left( \frac{1}{4} LK^6 C_{\max }^2\Vert F(x_{0})-y\Vert +1\right) } \right\} . \end{aligned}$$

Taking into account Step 1 and the updating rule at Step 2.4, it can be concluded that, at termination of Step 2, the trust-region radius $\Delta _k$ satisfies

$$\begin{aligned} \Delta _k\ge \min \left\{ C_{\min } , \, \gamma \omega \right\} \Vert B_k^{1/2} f'_k\Vert . \end{aligned}$$

In fact, in case of a smaller value of $\Delta _k$, it happens $f_\delta (x_k^\delta +p_k)\le \frac{1}{2}\Vert F(x_k^{\delta })-y^\delta +F'\left( x_k^{\delta }\right) p_k\Vert ^2$, then $\pi _k(p_k)\ge 1>\eta$, and the loop at Step 2 terminates. Then, it terminates for a trust-region radius greater than or equal to the one estimated above.

Then, $\lambda _k\le \bar{\lambda }$ as

$$\begin{aligned} \lambda _k\le \frac{\Vert B_k^{1/2} f'_k\Vert }{\Delta _k} \le \max \left\{ \frac{1}{ \gamma \omega },\, \frac{1}{C_{\min }} \right\} , \end{aligned}$$

and the thesis follows. $\square$

1.2 Proof of Theorem 4.3

Proof

Summing up from ${\bar{k}}$ to $k^*(\delta )-1$, by (37), (28), (10) and Lemma 4.4, it follows

$$\begin{aligned} (k^*(\delta )-{\bar{k}})\tau ^2 \delta ^2 \le \sum _{k={\bar{k}}}^{k^*(\delta )-1} \Vert F'\left( x_k^{\delta }\right) ^*(F\left( x_k^{\delta }\right) -y^{\delta })\Vert ^2\le \frac{\theta _{{\bar{k}}} {\bar{\lambda }}}{2(\theta _{{\bar{k}}}-1)q^2}\Vert x_{{\bar{k}}}^\delta -x^{\dagger }\Vert ^2. \end{aligned}$$

Thus, $k^*(\delta )$ is finite for $\delta >0$.

Convergence of $x^\delta _{k^*(\delta )}$ to a stationary point of (1) as $\delta$ tends to zero is obtained by adapting the proof of [5, Theorem 4.5]. Specifically, let $x^*$ be the limit of the sequence $\{x_k\}$ corresponding to the exact data y and let $\{\delta _n\}$ be a sequence of values of $\delta$ converging to zero as $n\rightarrow \infty$. Denote by $y^{\delta _n}$ a corresponding sequence of perturbed data, and by $k_n = k_*(\delta _n)$ the stopping index determined from the discrepancy principle (37) applied with $\delta =\delta _n$. Assume first that ${\tilde{k}}$ is a finite accumulation point of $\{k_n\}$. Without loss of generality, possibly considering a subsequence, it can be assumed that $k_n = {\tilde{k}}$ for all $n\in \mathbb {N}$. Thus, from the definition of $k_n$, it follows that

$$\begin{aligned} \Vert F'\left( x_{{\tilde{k}}}^{\delta _n}\right) ^* \left( y^{\delta _n}-F\left( x_{{\tilde{k}}}^{\delta _n}\right) \right) \Vert \le \tau \delta _n. \end{aligned}$$

(57)

As, by assumption, $\pi _k(x_{ k+1}-x_k)\ne \eta$, for all k, it follows that for the fixed index ${\tilde{k}}$, the iterate $x_{{\tilde{k}}}^{\delta }$ depends continuously on $\delta$. Then

$$\begin{aligned} x_{{\tilde{k}}}^{\delta _n}\rightarrow x_{{\tilde{k}}}, \qquad F'\left( x_{\tilde{k}}^{\delta _n}\right) \rightarrow F'(x_{{\tilde{k}}}), \qquad F\left( x_{\tilde{k}}^{\delta _n}\right) \rightarrow F(x_{{\tilde{k}}}) \qquad \text { as } \delta _n\rightarrow 0. \end{aligned}$$

Therefore, by (57), it follows that $F'(x_{{\tilde{k}}})^*(y-F(x_{{\tilde{k}}}))=0$, and the ${\tilde{k}}$th iterate with exact data y is a stationary point of (1), i.e. $x^* = x_{{\tilde{k}}}$, and it is possible to conclude that $x_{k_n}^{\delta _n}\rightarrow x^*$ as $\delta _n\rightarrow 0$.

It remains to consider the case where $k_n\rightarrow \infty$ as $n\rightarrow \infty$. As $\{x_k\}$ converges to a stationary point $x^*$ of (1) by Theorem 4.2, there exists $\tilde{k}>0$ such that

$$\begin{aligned} \Vert x_k -x^* \Vert \le \frac{1}{2}\bar{\rho } \qquad \text { for all } \qquad k\ge \tilde{k}, \end{aligned}$$

where $\bar{\rho }< \min \left\{ \displaystyle \frac{(q-\sigma )\tau -K(\sigma +1)}{c(K+\tau )}, \rho \right\}$. Then, as $x_k^\delta$ depends continuously on $\delta$, $\delta _n$ tends to zero and $k_*(\delta _n)\rightarrow \infty$, there exists $\delta _n$ sufficiently small such that ${\tilde{k}}\le k_*(\delta _n)$, and

$$\begin{aligned} \Vert x_{{\tilde{k}}}^{\delta _n} -x_{{\tilde{k}}} \Vert \le \frac{1}{2} \bar{\rho }. \end{aligned}$$

Then, for $\delta _n$ sufficiently small

$$\begin{aligned} \Vert x_{\tilde{k}}^{\delta _n}-x^*\Vert \le \Vert x_{\tilde{k}}^{\delta _n}- x_{\tilde{k}}\Vert + \Vert x_{\tilde{k}}-x^*\Vert \le \bar{\rho }. \end{aligned}$$

(58)

Now, from item (i) of Lemma 4.4, it follows $x_{\tilde{k}}^{\delta _n}\in \mathcal {B}_{2\rho }(x_{{\bar{k}}}^{\delta _n})$, while from (46) and Theorem 4.2 it holds $x^*\in \mathcal {B}_{2\rho }(x_{{\bar{k}}}^{\delta _n})$ as

$$\begin{aligned} \Vert x_{{\bar{k}}}^{\delta _n}-x^*\Vert \le \Vert x_{{\bar{k}}}^{\delta _n}-x^\dagger \Vert +\Vert x^\dagger -x^*\Vert \le 2\rho . \end{aligned}$$

Letting $e_k^*=x^*-x_k^{\delta _n}$. Repeating arguments from Lemma 4.3, and using (38), (2) it follows

$$\begin{aligned} \Bigg \Vert m_{{\tilde{k}}}\left( e^*_{{\tilde{k}}}\right) \Bigg \Vert\le & {} K\delta _n+ \Bigg \Vert F'\left( x_{\tilde{k}}^{\delta _n}\right) ^{*}\left( y -F\left( x_{\tilde{k}}^{\delta _n}\right) +F'\left( x_{\tilde{k}}^{\delta _n}\right) (x^*-x_{\tilde{k}}^{\delta _n})\right) \Bigg \Vert \\\le & {} K\delta _n+\left( c\Bigg \Vert x^*-x_{\tilde{k}}^{\delta _n}\Bigg \Vert +\sigma \right) \, \Bigg \Vert F'\left( x_{\tilde{k}}^{\delta _n}\right) ^{*} (y-F(x_{\tilde{k}}^{\delta _n}))\Bigg \Vert \\\le & {} (1+ c\Bigg \Vert x^*-x_{\tilde{k}}^{\delta _n}\Bigg \Vert +\sigma ) K\delta _n+ \left( c\Bigg \Vert x^*-x_{\tilde{k}}^{\delta _n}\Bigg \Vert +\sigma \right) \\&\Bigg \Vert F'\left( x_{\tilde{k}}^{\delta _n}\right) ^{*}\left( y^{\delta _n}-F\left( x_{\tilde{k}}^{\delta _n}\right) \right) \Bigg \Vert . \end{aligned}$$

Then, at iteration ${\tilde{k}}$, conditions (37) and (28) give

$$\begin{aligned} \Vert m_{{\tilde{k}}}\left( e^*_{{\tilde{k}}}\right) \Vert\le & {} \left( K\frac{1+ c\Vert x^*-x_{\tilde{k}}^{\delta _n}\Vert +\sigma }{\tau }+ \left( c\Vert x^*-x_{\tilde{k}}^{\delta _n}\Vert +\sigma \right) \right) \\&\Vert F'\left( x_{\tilde{k}}^{\delta _n}\right) ^{*}\left( y^{\delta _n}-F\left( x_{\tilde{k}}^{\delta _n}\right) \right) \Vert \\\le & {} \left( K\frac{1+ c\Vert x^*-x_{{\tilde{k}}}^{\delta _n}\Vert +\sigma }{q\tau }+ \frac{c\Vert x^*-x_{\tilde{k}}^{\delta _n}\Vert +\sigma }{q} \right) \Vert m_{\tilde{k}}(p_{\tilde{k}})\Vert . \end{aligned}$$

Thus, by (58) and ${\bar{\rho }} < \min \left\{ \displaystyle \frac{(q-\sigma )\tau -K(\sigma +1)}{c(K+\tau )}, \rho \right\}$, it follows that

$$\begin{aligned} \Vert m_{{\tilde{k}}}\left( e^*_{{\tilde{k}}}\right) \Vert \le \frac{1}{\theta _{\tilde{k}}}\Vert m_{{\tilde{k}}}(p_{\tilde{k}})\Vert \end{aligned}$$

is satisfied with $\theta _{\tilde{k}}=\displaystyle \frac{q\tau }{1+c(1+\tau )\bar{\rho }+\sigma (1+\tau )}>1$. Replacing $x^\dagger$ with $x^*$, (10) gives $\Vert x_{{\tilde{k}}+1}^{\delta _n} -x^*\Vert < \Vert x_{{\tilde{k}}}^{\delta _n} -x^*\Vert$ and repeating the above arguments, by induction monotonicity of the error $\Vert x_k^{\delta _n} -x^*\Vert$ for $\tilde{k}\le k \le k_n$ is obtained. Then

$$\begin{aligned} \Vert x_{k_n}^{\delta _n}-x^*\Vert < \Vert x_{{\tilde{k}}}^{\delta _n}-x^*\Vert \le {\bar{\rho }}. \end{aligned}$$

(59)

Finally, repeating the previous arguments, it can be shown that, for every $0\le \epsilon \le {\bar{\rho }}$, there exists $\bar{\delta }_\epsilon$ such that $\Vert x_{k_n}^{\delta _n}- x^*\Vert \le \epsilon$ for all $\delta _n\le \bar{\delta }_\epsilon$, i.e.

$$\begin{aligned} x_{k_n}^{\delta _n}\rightarrow x^* \qquad \text { as } \qquad \delta _n\rightarrow 0, \end{aligned}$$

and the thesis is proved. $\square$