A class of residual-based extended Kaczmarz methods for solving inconsistent linear systems☆
Introduction
Consider a large-scale linear system where the coefficient matrix , , and is the unknown vector. The Kaczmarz method [1] or the algebraic reconstruction technique (ART) [2] is one of the most popular solvers for solving linear systems of equations. At each iteration, the Kaczmarz method uses the cyclic rule to choose a row of the matrix and projects the current iteration onto the corresponding hyperplane. Let stand for the th row of the coefficient matrix and , hence, for the algebraic reconstruction technique (Kaczmarz) where ( mod )+1, is the Euclidean inner product and is the corresponding norm in . In the literature, there was empirical evidence that selecting the rows nonuniformly at random may be more effective than selecting the rows via Kaczmarz’s cyclic manner [3], [4], [5], [6]. While this randomized Kaczmarz (RK) method is thus quite appealing for applications, no guarantees of its rate of convergence have been known. In 2009 [7], Strohmer and Vershynin firstly proved that it converges with an expected exponential rate. Then, Bai and Wu in [8] discussed an improved estimate about the convergence rate of the randomized Kaczmarz method. Recently, Guan and Li explored an alternative version of the randomized Kaczmarz method to solve a consistent linear system in [9]. In the scheme, they chose each row of the coefficient matrix with probability proportional to the square of the Euclidean norm of the residual of each corresponding equation, which improves the convergence rate.
When the linear system (1.1) is inconsistent, the RK method does not converge to its least-squares solution , where represents the generalized inverse of matrix . Needell [10] proved that the randomized Kaczmarz estimate vector is (in the limit) within a fixed distance from the least-squares solution and also that this distance is proportional to the distance of from the column space of . To solve this convergence problem, inspired by [11], [12], Zouzias and Freris proposed the randomized extended Kaczmarz (REK) method which exponentially converges in expectation to the least-squares solution in [13]. While, when the coefficient matrix is of full column rank, all its columns are independent of each other. If some of its columns are missed in the orthogonal projection process, the vector obtained in the second component of the REK method may not converge to , so that the iteration sequence of the REK method may not converge to (for more details see the example in [14]). Based on [8] and [13], Bai and Wu proposed a partially randomized extended Kaczmarz method (PREK) with a partial randomization strategy and proved the convergence, and derived an upper bound for the expected convergence rate under certain conditions in [14].
While, if is a constant, the PREK method selects one row of matrix with equal probability in the first component, then its advantage is lost. In this paper, motivated by [9], [14], for solving the inconsistent system (1.1) we present a new partially randomized extended Kaczmarz method, where the probability in the first component in the PREK method is to do with the residual of each corresponding equation. In this case, when each row of matrix has the same norm, the new method ensures that large entries of the residual vector should be preferentially annihilated. Meanwhile, since all its columns can be selected, the vector obtained in the second component of the new method can converge to , i.e., the iteration sequence converges to , even though the coefficient matrix is of full column rank.
Then, we establish the corresponding convergence theory, which proves that under certain conditions, the estimated upper bound of this method is less than or equal to that of the PREK method. Moreover, considering the various selection rules about the residuals in [11], [12], [15], we construct the other three extended Kaczmarz methods to solve the inconsistent system (1.1) and give the corresponding convergence analysis. The numerical results confirm the new four methods have advantages over the REK and PREK methods in terms of the number of iteration steps and calculation times in some situation.
The rest of this paper is organized as follows. In Section 2, we review the REK method, the PREK method, and some basic results. In Section 3, we focus on the PREKR method and establish its convergence theory. Furthermore, we propose the other three extended methods and give the convergence analysis. In Section 4, some numerical experiments are reported to show the effectiveness of the proposed algorithms. Finally, we present some conclusions in Section 5.
Section snippets
Preliminaries and notations
For convenience, we introduce some notations. For an real matrix , , , det(), and denote its transpose, Frobenius norm, determinant, the range and null spaces. At the same time, is the orthogonal complement space of . Define the condition number of as , where and are the smallest nonzero and largest eigenvalues of the matrix . For the constant , and represents the largest integer which is smaller than or
The new extended Kaczmarz methods and convergence analysis
Now, we propose a new randomized strategy for the first component in the PREK method, that is, the probability of each row of the coefficient matrix is proportional to the square of the Euclidean norm of the residual of each corresponding equation rather than the Euclidean norm of each row of matrix , and the orthogonal projections in the second component are in the given cyclic order. Thus, a new partially random extended Kaczmarz method with residuals (PREKR) is established.
Before proving
Numerical experiments
In this section, some numerical examples are provided to illustrate the PREKR, PREKRH, MREK and MDEK methods for solving the inconsistent linear system (1.1). All experiments are carried out using MATLAB (version R2020b) on a laptop with 2.20-GHZ intel Core i7-10870H processor, 16 GB memory, and Windows 10 operating system. We consider experiments on three types of matrices: (i) Random matrix; (ii) Random orthogonal matrix; (iii) Real-world matrix. For the first and third types of matrices, we
Conclusions
In this paper, we propose four new iteration schemes for solving inconsistent linear systems. The convergence analyses of the new methods are given. The numerical results from Example 4.1, Example 4.2, Example 4.3, Example 4.4 demonstrate clearly that the new methods outperform the REK and PREK methods in terms of both iteration counts and computation times.
For solving large-scale systems without the full column rank and overdetermined assumptions, the four new methods with random sampling are
CRediT authorship contribution statement
Wendi Bao: Conceptualization, Methodology. Zhonglu Lv: Writing – original draft. Feiyu Zhang: Visualization, Software. Weiguo Li: Review, Conceptualization.
Acknowledgments
The authors are thankful to the referees for their constructive comments and valuable suggestions, which greatly improved the original manuscript of this paper.
References (20)
- et al.
Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography
J. Theoret. Biol.
(1970) - et al.
On convergence rate of the randomized Kaczmarz method
Linear Algebra Appl.
(2018) - et al.
On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems
Linear Algebra Appl.
(2019) - et al.
On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems
Appl. Math. Lett.
(2018) Angenäherte auflösung von systemen linearer gleichungen
Bull. Int. Acad. Polon. Sci. Lett.
(1937)- et al.
Algebraic reconstruction techniques can be made computationally efficient
IEEE Trans. Med. Imag.
(1993) - et al.
New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling
Proc. SPIE - Int. Soc. Opt. Eng.
(1992) The Mathematics of Computerized Tomography
(1986)Convergence rates of the Kaczmarz-Tanabe method for linear systems
J. Comput. Appl. Math.
(2021)- et al.
A randomized Kaczmarz algorithm with exponential convergence
J. Fourier Anal. Appl.
(2009)
Cited by (0)
- ☆
This research is supported by the Fundamental Research Funds for the Central Universities (grant number 18CX02041A), the Fundamental Research Funds for the Central Universities (grant number 20CX05011A) and the Fundamental Research Funds for the Central Universities (grant number 19CX05003A-2)