A class of residual-based extended Kaczmarz methods for solving inconsistent linear systems

https://doi.org/10.1016/j.cam.2022.114529Get rights and content

Highlights

  • Four new extended Kaczmarz methods based on residuals are proposed.

  • For the new methods, we provide convergence analyses in terms of expectation.

  • Numerical experiments are given to show the efficiency of new methods.

Abstract

In this paper, we propose four extended Kaczmarz methods: two partially randomized methods like probability proportional to residual or residual homogenizing, and two deterministic strategies with the maximal-residual control and the maximum-distance control. Without the full column rank and overdetermined assumptions on linear systems, we provide a thorough convergence analysis in terms of expectation, and derive upper bounds for the expected convergence rates of the new extended Kaczmarz methods. Numerical experiments on Gaussian models as well as 2D image reconstruction problems demonstrate that the new extended Kaczmarz methods can be much more effective than the existing ones. Especially, the improvements of two deterministic strategies are very prominent.

Introduction

Consider a large-scale linear system Ax=b,where the coefficient matrix ARm×n, bRm, and xRn is the unknown vector. The Kaczmarz method [1] or the algebraic reconstruction technique (ART) [2] is one of the most popular solvers for solving linear systems of equations. At each iteration, the Kaczmarz method uses the cyclic rule to choose a row of the matrix and projects the current iteration onto the corresponding hyperplane. Let A(i) stand for the ith row of the coefficient matrix A and b=(b1,b2,,bm)T, hence, for the algebraic reconstruction technique (Kaczmarz) xk=xk1+biA(i),xk1A(i)22(A(i))T,k=1,2,,where i= (k mod m)+1, , is the Euclidean inner product and 2 is the corresponding norm in Rn. In the literature, there was empirical evidence that selecting the rows nonuniformly at random may be more effective than selecting the rows via Kaczmarz’s cyclic manner [3], [4], [5], [6]. While this randomized Kaczmarz (RK) method is thus quite appealing for applications, no guarantees of its rate of convergence have been known. In 2009 [7], Strohmer and Vershynin firstly proved that it converges with an expected exponential rate. Then, Bai and Wu in [8] discussed an improved estimate about the convergence rate of the randomized Kaczmarz method. Recently, Guan and Li explored an alternative version of the randomized Kaczmarz method to solve a consistent linear system in [9]. In the scheme, they chose each row of the coefficient matrix A with probability proportional to the square of the Euclidean norm of the residual of each corresponding equation, which improves the convergence rate.

When the linear system (1.1) is inconsistent, the RK method does not converge to its least-squares solution x=Ab, where A represents the generalized inverse of matrix A. Needell [10] proved that the randomized Kaczmarz estimate vector is (in the limit) within a fixed distance from the least-squares solution and also that this distance is proportional to the distance of b from the column space of A. To solve this convergence problem, inspired by [11], [12], Zouzias and Freris proposed the randomized extended Kaczmarz (REK) method which exponentially converges in expectation to the least-squares solution in [13]. While, when the coefficient matrix A is of full column rank, all its columns are independent of each other. If some of its columns are missed in the orthogonal projection process, the vector obtained in the second component of the REK method may not converge to bR(A), so that the iteration sequence {xk}k=0 of the REK method may not converge to x (for more details see the example in [14]). Based on [8] and [13], Bai and Wu proposed a partially randomized extended Kaczmarz method (PREK) with a partial randomization strategy and proved the convergence, and derived an upper bound for the expected convergence rate under certain conditions in [14].

While, if A(i)2(i=1,2,,m) is a constant, the PREK method selects one row of matrix A with equal probability in the first component, then its advantage is lost. In this paper, motivated by [9], [14], for solving the inconsistent system (1.1) we present a new partially randomized extended Kaczmarz method, where the probability in the first component in the PREK method is to do with the residual of each corresponding equation. In this case, when each row of matrix A has the same norm, the new method ensures that large entries of the residual vector should be preferentially annihilated. Meanwhile, since all its columns can be selected, the vector obtained in the second component of the new method can converge to R(A), i.e., the iteration sequence {xk}k=0 converges to x, even though the coefficient matrix A is of full column rank.

Then, we establish the corresponding convergence theory, which proves that under certain conditions, the estimated upper bound of this method is less than or equal to that of the PREK method. Moreover, considering the various selection rules about the residuals in [11], [12], [15], we construct the other three extended Kaczmarz methods to solve the inconsistent system (1.1) and give the corresponding convergence analysis. The numerical results confirm the new four methods have advantages over the REK and PREK methods in terms of the number of iteration steps and calculation times in some situation.

The rest of this paper is organized as follows. In Section 2, we review the REK method, the PREK method, and some basic results. In Section 3, we focus on the PREKR method and establish its convergence theory. Furthermore, we propose the other three extended methods and give the convergence analysis. In Section 4, some numerical experiments are reported to show the effectiveness of the proposed algorithms. Finally, we present some conclusions in Section 5.

Section snippets

Preliminaries and notations

For convenience, we introduce some notations. For an m×n real matrix A, AT, AF, det(A), R(A) and N(A) denote its transpose, Frobenius norm, determinant, the range and null spaces. At the same time, R(A) is the orthogonal complement space of R(A). Define the condition number of A as κ(A)λmax(ATA)λmin(ATA), where λmin(ATA) and λmax(ATA) are the smallest nonzero and largest eigenvalues of the matrix ATA. For the constant c, c and |c| represents the largest integer which is smaller than or

The new extended Kaczmarz methods and convergence analysis

Now, we propose a new randomized strategy for the first component in the PREK method, that is, the probability of each row of the coefficient matrix A is proportional to the square of the Euclidean norm of the residual of each corresponding equation rather than the Euclidean norm of each row of matrix A, and the orthogonal projections in the second component are in the given cyclic order. Thus, a new partially random extended Kaczmarz method with residuals (PREKR) is established.

Before proving

Numerical experiments

In this section, some numerical examples are provided to illustrate the PREKR, PREKRH, MREK and MDEK methods for solving the inconsistent linear system (1.1). All experiments are carried out using MATLAB (version R2020b) on a laptop with 2.20-GHZ intel Core i7-10870H processor, 16 GB memory, and Windows 10 operating system. We consider experiments on three types of matrices: (i) Random matrix; (ii) Random orthogonal matrix; (iii) Real-world matrix. For the first and third types of matrices, we

Conclusions

In this paper, we propose four new iteration schemes for solving inconsistent linear systems. The convergence analyses of the new methods are given. The numerical results from Example 4.1, Example 4.2, Example 4.3, Example 4.4 demonstrate clearly that the new methods outperform the REK and PREK methods in terms of both iteration counts and computation times.

For solving large-scale systems without the full column rank and overdetermined assumptions, the four new methods with random sampling are

CRediT authorship contribution statement

Wendi Bao: Conceptualization, Methodology. Zhonglu Lv: Writing – original draft. Feiyu Zhang: Visualization, Software. Weiguo Li: Review, Conceptualization.

Acknowledgments

The authors are thankful to the referees for their constructive comments and valuable suggestions, which greatly improved the original manuscript of this paper.

References (20)

There are more references available in the full text version of this article.

Cited by (0)

This research is supported by the Fundamental Research Funds for the Central Universities (grant number 18CX02041A), the Fundamental Research Funds for the Central Universities (grant number 20CX05011A) and the Fundamental Research Funds for the Central Universities (grant number 19CX05003A-2)

View full text