Convergence of Gauss–Newton’s method and uniqueness of the solution

doi:10.1016/j.amc.2004.12.055

Applied Mathematics and Computation

Volume 170, Issue 1, 1 November 2005, Pages 686-705

https://doi.org/10.1016/j.amc.2004.12.055 Get rights and content

Abstract

In this paper, we study the convergence of Gauss–Newton’s method for nonlinear least squares problems. Under the hypothesis that derivative satisfies some kinds of weak Lipschitz condition, we obtain the exact estimates of the radii of convergence ball of Gauss–Newton’s method and the uniqueness ball of the solution. New results can be used to determinate approximation zero of Gauss–Newton’s method.

Introduction

We consider the nonlinear least squares problems: $\min F (x) : = \frac{1}{2} f (x)^{T} f (x)$ where f(x): Rⁿ → R^m is Frechet differentiable, m ⩾ n. And in all cases ∥ · ∥ refers to the 2-norm.

Gauss–Newton’s method, defined by $x_{n + 1} = x_{n} - {[f^{'} (x_{n})^{T} f^{'} (x_{n})]}^{- 1} f^{'} (x_{n})^{T} f (x_{n}), n ⩾ 0$ is one of the best-known methods for the problem (1.1). For the Gauss–Newton’s method, analyzes for local and rate of convergence properties are mostly restricted in quality [1], [6], [15], [16], only considering the existence neighborhood of the convergence, but it cannot make us clearly see how big the radius of the convergence ball. Let x* denote the solution of (1.1), $B (x, r)$ denote an open ball with radius r and center x, and let $\bar{B (x, r)}$ denote its closure.

Traub and Wozniakowski [5] and Wang [12] independently gave an exact estimate for the convergence ball of Newton’s method. Under the hypothesis that f′(x) satisfies the some kind Lipschitz condition $‖ f^{'} (x^{*})^{- 1} (f^{'} (x) - f^{'} (x^{τ})) ‖ ⩽ \int_{τ ρ (x)}^{ρ (x)} L (u) d u, \forall x \in B (x^{*}, r),$ where ρ(x) = ∥x − x*∥, x^τ = x* + τ(x − x*), and L is monotonic function, Wang [13], [14] studied the convergence of the Newton’s method.

In this paper, we consider the convergence of the Gauss–Newton’s method. Under more general conditions, we obtain the convergence domain of the Gauss–Newton’s method and the uniqueness domain of the solution. Further, we can prove the optimality of the estimation of the radii. New results can be used to determinate approximation zero of Gauss–Newton’s method.

Section snippets

Special and generalized Lipschitz condition

The condition on the function f(x) $‖ f (x) - f (x^{τ}) ‖ ⩽ L ‖ x - x^{τ} ‖, \forall x \in B (x^{*}, r),$ where x^τ = x* + τ(x − x*),0 ⩽ τ ⩽ 1, is usually called radius Lipschitz condition in the ball B(x*, r) with constant L. Sometimes if it is only required to satisfy $‖ f (x) - f (x^{*}) ‖ ⩽ L ‖ x - x^{*} ‖, \forall x \in B (x^{*}, r),$ we call it the center Lipschitz condition in the ball B(x*, r) with constant L. Furthermore, the L in the Lipschitz condition need not be a constant, but a positive integrable function, If this is the case, then (2.1) or (2.2) is replaced by $‖ f (x) - f (x^{τ}$

Convergence ball of Gauss–Newton’s method

Theorem 3.1

Suppose x* satisfies (1.1), f has a continuous derivative in B(x*,r). f′(x*) has full rank, and f′ satisfies the radius Lipschitz condition with L average. $‖(f^{'} (x) - f^{'} (x^{τ}))‖ ⩽ \int_{τ ρ (x)}^{ρ (x)} L (u) d u, \forall x \in B (x^{*}, r), 0 ⩽ τ ⩽ 1,$ where x^τ = x* + τ(x − x*), ρ(x) = ∥ x − x*∥, and L is nondecreasing. Let r > 0 satisfy $\frac{β \int_{0}^{r} L (u) u d u}{r (1 - β \int_{0}^{r} L (u) d u)} + \frac{\sqrt{2} c β^{2} \int_{0}^{r} L (u) d u}{r (1 - β \int_{0}^{r} L (u) d u)} ⩽ 1 .$ Then Gauss–Newton’s method is convergent for all x₀ ∈ B(x*,r) and $‖ x_{n + 1} - x^{*} ‖ ⩽ \frac{β \int_{0}^{ρ (x_{0})} L (u) u d u}{ρ (x_{0})^{2} (1 - β \int_{0}^{ρ (x_{0})} L (u) d u)} ‖ x_{n} - x^{*} ‖^{2} + \frac{\sqrt{2} c β^{2} \int_{0}^{ρ (x_{0})} L (u) d u}{ρ (x_{0}) (1 - β \int_{0}^{ρ (x_{0})} L (u) d u)} ‖ x_{n} - x$

The uniqueness ball for the optimal solution

Theorem 4.1

Suppose x* satisfies (1.1), f has a continuous derivative in B(x*,r). f′(x*) has full rank, and f′ satisfies the center Lipschitz condition with L average. $‖ f^{'} (x) - f^{'} (x^{*}) ‖ ⩽ \int_{0}^{ρ (x)} L (u) d u, \forall x \in B (x^{*}, r),$ where ρ(x) = ∥x − x*∥, and L is nondecreasing. Let r > 0 satisfy $\frac{β}{r} \int_{0}^{r} L (u) (r - u) d u + \frac{c β_{0}}{r} \int_{0}^{r} L (u) d u ⩽ 1,$ where c,β hold in (3.4), and $β_{0} = ‖{[f^{'} (x^{*})^{T} f^{'} (x^{*})]}^{- 1}‖ .$ Then Eq. (1.1) has a unique solution x* in B(x*,r).

Proof

Suppose x₀ ∈ B(x*,r), x₀ ≠ x* is also a solution of (1.1). Then we have ${[f^{'} (x^{*})^{T} f^{'} (x^{*})]}^{- 1} f^{'} (x_{0})^{T} f (x_{0}) = 0 .$

Hence $x_{0} - x^{*} = x_{0} - x^{*} - [f^{'} (]$

The optimality of the estimation of the radius

Theorem 5.1

Suppose that the equality sign holds in the inequality (3.2) in the Theorem 3.1. Then the given value r of the convergence ball is the best possible.

Proof

We notice that when r is determined by equality $\frac{β \int_{0}^{r} L (u) u d u}{r (1 - β \int_{0}^{r} L (u) d u)} + \frac{\sqrt{2} c β^{2} \int_{0}^{r} L (u) u d u}{r (1 - β \int_{0}^{r} L (u) d u)} = 1,$ there exists f satisfying (3.1) in B(x*, r) and x₀ on the boundary of the closed ball such that Gauss–Newton’s method fails. In fact, the following is an example on the scaled case: $f (x) = \{\begin{matrix} x^{*} - x + β \int_{0}^{x - x^{*}} (x - x^{*} - u) L (u) d u, x^{*} ⩽ x < x^{*} + r; \\ x^{*} - x + β \int_{0}^{x^{*} - x} (x - x^{*} + u) L (u \end{matrix}$

Corollaries of the main results

In the study of the Gauss–Newton’s method (or Newton’s method), the assumption that the derivative is Lipschitz continuous is considered traditional. Combining Theorem 3.1, Theorem 4.1 with Theorem 5.1, Theorem 5.2, and taking L as a constant, the following two corollaries are obtained directly.

Corollary 6.1

Suppose x* satisfies (1.1), f has a continuous derivative in B(x*,r). f′(x*) has full rank, and f′ satisfies the radius Lipschitz condition: $‖ f^{'} (x) - f^{'} (x^{τ}) ‖ ⩽ (1 - τ) L ‖ x^{τ} - x^{*} ‖, \forall x \in B (x^{*}, r), 0 ⩽ τ ⩽ 1,$ where x^τ = x* + τ(x − x*

Convergence under weaker Lipschitz condition

In this section, we consider the (1.2) under more weaker Lipschitz condition.

Theorem 7.1

Suppose x* satisfies (1.1), f(x*) = 0, f has a continuous derivative in B(x*,r). f′(x*) has full rank, and f′ satisfies the radius Lipschitz condition with L average. $‖ f^{'} (x)^{†} (f^{'} (x) - f^{'} (x^{τ})) ‖ ⩽ \int_{τ ρ (x)}^{ρ (x)} L (u) d u, \forall x \in B (x^{*}, r), 0 ⩽ τ ⩽ 1,$ $f^{'} (x) hasfullrank, \forall x \in B (x^{*}, r),$ where x^τ = x* + τ(x − x*), ρ(x) = ∥x − x*, f′(x)^† = [f′(x)^T f′(x)]⁻¹ f′(x)^T, and L is nondecreasing. Let r > 0 satisfy $\frac{\int_{0}^{r} L (u) u d u}{r} ⩽ 1 .$ Then Gauss–Newton’s method is convergent for all x₀ ∈ B(x*

Applications to determination of an approximation zero

To study the property of quadratic convergence of Newton’s method and computational complexity of zeros, Smale [8], [9], [10] proposed the definitions of approximation zeros of Newton’s method. With the Smale’s studies, we can propose a new definition of the approximation zeros for the Gauss–Newton’s method. This definition is as follows.

Definition 8.1

If x₀∈Rⁿ such that Gauss–Newton’s method (1.2) for f(x):Rⁿ → R^m is well defined and (3.6) is satisfied with $q = \frac{1}{2}$ , then x₀ is called an approximation zero of the

Remark

In fact, the results in this paper can be generalized in real or complex infinite dimensional Hilbert space, we will continue to further study for the convergence of Gauss–Newton’s method in real or complex infinite dimensional Hilbert space.

Acknowledgement

Supported by the Financially- Aiding Program for the Backbone Teachers of Ministry of Education of China and Natural Science Foundation of University of Petroleum.

References (16)

T. Yamamoto
Historical developments in convergence analysis for Newton’s and Newton-like methods
J. Comput. Appl. Math.
(2000)
I.k. Argyros
On Newton’s method under mild differentiability conditions and applications
Appl. Math. Comput.
(1999)
G.W. Stewart
On the continuity of the generalized inverse
SIAM J. Appl. Math.
(1960)
I.K. Argyros
The Newton-Kantorovich method under mild differentiability conditions and the Ptak estimates
Monatsch. Math.
(1990)
J.F. Traub et al.
Covergence and complexity of Newton iteration
J. Assoc. Comput. Math.
(1979)
J.M. Ortega et al.
Iterative Solution of Nonlinear Equations in Several Variables
(1970)
P.A. Wedin
Perturbation theory for pseudo-inverse
BIT
(1973)
S. Smale
The fundamental theorem of algebra and complexity theory
Bull. Amer. Math. Soc.
(1981)

There are more references available in the full text version of this article.

Cited by (43)

An efficient iterative optimization-based algorithm for the real-time estimation of harmonics under power system frequency deviations
2023, Engineering Science and Technology, an International Journal
In current electrical power industry, a great number of power electronic devices produce harmonics. Moreover, the fundamental system frequency of a power line can deviate in the cases of fluctuating and periodically varying loads and transients. This frequency deviation introduces further undesirable frequency components in the spectrum. Correct measurement of harmonics under such system frequency deviations has prominent importance for the protection of power electronics devices. For this goal, this paper proposes a new iterative optimization-based algorithm for the estimation of harmonics in these circumstances. The method is based on time-domain Gauss-Newton optimization of distorted power line signal which attains its parameters directly from the spectrum of the signal by employing local extrema of the corresponding spectrum. In the first phase of the algorithm, the deviated system frequency is estimated. In the second phase, the harmonic components are estimated and fine-tuned correspondingly. Case studies show that the proposed method is robust and precisely estimates deviations in the system’s fundamental frequency and measure the amplitudes of harmonic components very accurately. It requires just a stoppage criterion. The computational assessment of the method also shows its time efficiency in identifying harmonics. Thus, it is a very good candidate for real-time harmonic measurements.
Cycle life prediction of NiCo<inf>2</inf>O<inf>4</inf>//activated carbon asymmetric supercapacitors
2022, Journal of Energy Storage
Citation Excerpt :
Combined with the initial life values of the acceleration experiments in Table 1, the weight of each influencing factor was determined by the method of pairwise comparison. And then, the model Eq. (6) was established by the least square method [51–54] and further optimized by the Gauss-Newton method [55,56]. At the same time, JMP software was applied to solve the model Eq. (6) to obtain the parameters of A, B, Z, and n. Subsequently, the model Eq. (6) obtained above was verified by the aging test under standard conditions in Table 2.
Here, we report that the NiCo₂O₄ nanoflower-spherical material, which has a specific capacitance of 1503 F g⁻¹ at the current density of 1 A g⁻¹, is synthesized by the hydrothermal method. And asymmetric supercapacitors (ASCs) of NiCo₂O₄//activated carbon (AC) exhibit the energy density of 25.02 Wh kg⁻¹ at a power density of 760 W kg⁻¹. Furthermore, to investigate the impact of aging factors on the cycle life of ASCs, a new model for capacitance life prediction is established. The model takes the cycle times, temperature, current density, and potential window into account, which is based on the classic Eyring's law and the actual specific capacitance of ASCs. It forecasts that the ASCs will have a cycle life of 9750 cycles under ideal condition of 25 °C, 5 A g⁻¹ current density, and 0–1.5 V potential range, with only a relative error of −0.09 % and a root mean square error (RMSE) of 5.47 %, compared to the actual life of 9759 cycles. Therefore, this model provides a strong basis for the life prediction and optimization of asymmetric supercapacitors.
Local convergence of the Gauss–Newton method for injective-overdetermined systems of equations under a majorant condition
2013, Computers and Mathematics with Applications
Citation Excerpt :
In addition to the special cases obtained in [5], the lack of convexity of the derivative of the majorant function in this analysis, allows us to obtain two new important special cases, namely, the convergence can be ensured under Hölder-like and generalized Lipschitz conditions. In the latter case, the results are obtained without assuming that the function that defines the condition is nondecreasing, thus generalizing Corollary 6.3 in [8]. Moreover, it is worth to mention that, similar to the convergence analysis of the Newton method (see [2]), the hypothesis of convex derivative of the majorant function or nondecreasing of the function which defines the generalized Lipschitz condition, are needed only to obtain quadratic convergence rate.
A local convergence analysis of the Gauss–Newton method for solving injective-overdetermined systems of nonlinear equations under a majorant condition is provided. The convergence as well as results on its rate are established without a convexity hypothesis on the derivative of the majorant function. The optimal convergence radius, the biggest range for uniqueness of the solution along with some other special cases are also obtained.
Improved local convergence analysis of inexact Gauss-Newton like methods under the majorant condition in Banach spaces
2013, Journal of the Franklin Institute
Citation Excerpt :
The semilocal convergence matter is, based on the information around an initial point, to give criteria ensuring the convergence of iterative procedures; while the local one is, based on the information around a solution, to find estimates of the radii of convergence balls. A plethora of sufficient conditions for the local as well as the semilocal convergence of Newton-type methods as well as an error analysis for such methods can be found in [1–14,16–22]. Argyros [4] and Argyros et al. [7], Banach's Lemma
We present a local convergence analysis of inexact Gauss–Newton like methods for solving nonlinear equations in a Banach space setting. Using more precise majorant conditions than in earlier studies, we provide a larger radius of convergence; tighter error estimates on the distances involved and a clearer relationship between the majorant function and the associated least squares problem. Moreover, these advantages are obtained under the same computational cost.
A Gauss–Newton method for mixed least squares-total least squares problems
2024, Calcolo
A Robust and High-Precision Three-Step Positioning Method for an Airborne SAR Platform
2023, Remote Sensing

View all citing articles on Scopus

View full text

Convergence of Gauss–Newton’s method and uniqueness of the solution

Abstract

Introduction

Section snippets

Special and generalized Lipschitz condition

Convergence ball of Gauss–Newton’s method

The uniqueness ball for the optimal solution

The optimality of the estimation of the radius

Corollaries of the main results

Convergence under weaker Lipschitz condition

Applications to determination of an approximation zero

Remark

Acknowledgement

J. Comput. Appl. Math.

Appl. Math. Comput.

On the continuity of the generalized inverse

SIAM J. Appl. Math.

The Newton-Kantorovich method under mild differentiability conditions and the Ptak estimates

Monatsch. Math.

Covergence and complexity of Newton iteration

J. Assoc. Comput. Math.

Iterative Solution of Nonlinear Equations in Several Variables

Perturbation theory for pseudo-inverse

BIT

The fundamental theorem of algebra and complexity theory

Bull. Amer. Math. Soc.