Directory UMM :Data Elmu:jurnal:J-a:Journal of Computational And Applied Mathematics:Vol100.Issue1.1998:
Journal of Computational and Applied Mathematics 100 (1998) 1–9
Block SOR methods for rank-de cient least-squares problems
C.H. Santos ∗ , B.P.B. Silva, J.Y. Yuan 1
Departamento de Matematica - UFPR, Centro Politecnico, CP: 19.081, CEP: 81531-990, Curitiba, Parana, Brazil
Received 31 August 1997; received in revised form 2 July 1998
Abstract
Many papers have discussed preconditioned block iterative methods for solving full rank least-squares problems. However very few papers studied iterative methods for solving rank-de cient least-squares problems. Miller and Neumann
(1987) proposed the 4-block SOR method for solving the rank-de cient problem. Here a 2-block SOR method and a
3-block SOR method are proposed to solve such problem. The convergence of the block SOR methods is studied. The
optimal parameters are determined. Comparison between the 2-block SOR method and the 3-block SOR method is given
c 1998 Elsevier Science B.V. All rights reserved.
also.
AMS classi cation: 65F10
Keywords: Preconditioned iterative method; Block SOR method; Convergence; Least-squares problem; Rank-de cient
least-squares problem; Optimal parameter
1. Introduction
We frequently meet rank-de cient least-squares problems de ned by
minn kAx − bk2 ;
x∈r
(1.1)
where A is m×n matrix with m¿n and rank (A) ¡ n, when we solve the actual problems in statistics,
economics, genetics, di erential equations, and image and signal processing. Recently, least-squares
methods have got more attention in application areas, and also in applied mathematics society.
Many papers have studied the solvers of the rank-de cient least-squares problems. The general
and ecient ways are singular-value decomposition and QR decomposition. It is well-known that the
iterative methods are preferable for large sparse problems. For rank-de cient least-squares problems,
∗
1
Corresponding author.
Partially supported by grant 301035/93-8 of CNPq, Brazil.
c 1998 Elsevier Science B.V. All rights reserved.
0377-0427/98/$ – see front matter
PII: S 0 3 7 7 - 0 4 2 7 ( 9 8 ) 0 0 1 1 4 - 9
2
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
there are very few papers to study the iterative methods for solving rank-de cient problems. Miller
and Neumann discussed the SOR method to solve the problem (1987) [6]. They partitioned the
matrix A into four parts and then applied the SOR method to solve the new system. In fact, the big
problem for iterative methods to solve the rank-de cient problems is the determination of the rank
of the matrix A. The problem is simple in theory, but not in applications.
Recently, Bjorck and Yuan [2] proposed three algorithms to nd linearly independent rows of
the matrix A by LU factorization, Luo, et al. [4] used the basic solution method (Benzi and Meyer
called direct-projection method [1]) to nd rank of A, and Silva and Yuan [10] applied the QR
decomposition in column-wise to nd the set of linearly independent rows of A. Those methods
motivate us to consider iterative methods for solving the rank-de cient problems. We can nd the
desired preconditioner for rank-de cient least-squares problems by their algorithms.
Since there are many solutions for the rank-de cient problems and we are generally interested in
just the minimum 2-norm solution, we shall derive a new system for the minimum 2-norm solution.
We shall study block SOR methods to solve the new equation by preconditioning technique. Our
block SOR methods are di erent from Miller and Neumann’s SOR method. We shall study the
convergence and optimal relaxation parameter of the block SOR methods, and give some comparison
result between our block SOR methods. Like full rank case, we show the 2-block SOR method always
converges for certain value of the relaxation parameter.
The outline of this paper is as follows. The new system of normal equation for the rank-de cient
least-squares problems is derived in Section 2. The 3-block SOR method is proposed in Section 3.
Its convergence and optimal parameter are also discussed in the section. The convergence theory
shows the 3-block SOR method just converges for some certain case with certain conditions. In
Section 4, the 2-block SOR method is studied. The convergence of the 2-block SOR method tells
us the 2-block SOR method is always convergent for rank-de cient problems with certain relaxation
parameter like the 2-block SOR method for full rank problems. The optimal parameter for the 2block SOR method is given as well in Section 4. The comparison between the 2-block SOR method
and the 3-block SOR method is investigated. The result shows the 2-block SOR method is better
than the 3-block SOR method for rank-de cient problems as full rank problems. Throughout the
paper, we always assume that the matrix A with rank (A) = k ¡ n has the partition
A=
A1
A2
(1.2)
;
where A1 ∈ Rk×n is full row rank, and A2 ∈ R(m−k)×n .
2. New system of normal equation
For the treatment of problem (1.1), we need the following lemma.
Lemma 2.1. Assume that the matrix A has the structure of (1.2). Then
N(A) = N(A1 )
and
R(AT ) = R(AT1 );
where R(A) and N (A) are range and null of A; respectively.
(2.1)
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
3
Proof. ∀x ∈ N (A), it follows from Ax = 0 that A1 x = 0, that is x ∈ N (A1 ). Hence
N (A) ⊂ N (A1 ):
(2.2)
Since A has the structure of (1.2) and rank (A) = k, there exists one nonsingular matrix P such that
PA =
A1
0
(2.3)
:
∀x ∈ N (A1 ), it follows from A1 x = 0 and (2.3) that PAx = 0, that is x ∈ N (A). Thus
N (A1 ) ⊂ N (A):
(2.4)
We have shown N (A) = N (A1 ) by combining (2.2) and (2.4). In terms of N (A) = N (A1 ), it is
obvious that R(AT ) = R(AT1 ).
Since the minimum 2-norm solution x of the rank-de cient least-squares problem of (1.1) is in
R(AT ), that is in R(AT1 ) by Lemma 2.1, We can consider the transformation
x = AT1 y;
(2.5)
where y ∈ Rk , to obtain the minimum 2-norm solution of the problem of (1.1). Substituting (2.5)
into (1.1), we obtain the new system of the normal equation of the problem (1.1) for rank-de cient
case as follows:
A1 AT AAT1 y = A1 AT b:
(2.6)
By the structure of A in (1.2), we can rewrite the system (2.6) as an augmented form
or
AAT1
0
A1 AT1
I
A1 AT
A2 A T
1
0
y
r
0
I
I
0
A1 AT2
A1 AT1
where
r = b − Ax = b −
=
b
0
(2.7)
;
y
b1
r2 = b2 ;
r1
0
AAT1 y
=
r1
r2
;
b=
(2.8)
b1
b2
have the same partition as A in (1.2). Therefore we have shown the following theorem.
Theorem 2.2. Suppose that the matrix A in the problem of (1.1) has the partition of (1.2) with
rank (A) = k = rank (A1 ) ¡ n. Then the minimum 2-norm solution x of the rank-de cient leastsquares problems of (1.1) is given by
x = AT1 y;
where y is the unique solution of the system of (2.7) or (2.8).
4
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
Moreover, the general solution x of the rank-de cient problem of (1.1) is given by
x = AT1 y + z;
∀z ∈ N (A1 ):
The remaining of the paper will discuss block SOR methods to solve the system (2.8). For the
SOR method, there is a well-known eigenvalue functional between the Jacobi iteration matrix and
the SOR iteration matrix.
Theorem 2.3. (Young [11]). Assume that the Jacobi iteration matrix J is weakly p-cyclic and
consistently ordered. Suppose is eigenvalue of J . If satis es the functional
( + ! − 1)p = !p p p−1 ;
(2.9)
then is eigenvalue of the SOR iteration matrix L! . On the contrary, if is eigenvalue of L! ,
and satis es (2.9), then is eigenvalue of J .
3. The 3-block SOR method
3.1. The 3-block SOR algorithm
Consider the 3-block diagonal matrix D = diag (A1 AT1 ; I; A1 AT1 ) for the system of (2.8). We can
obtain the 3-block SOR method as follows:
y(l+1)
y(l)
A1 AT1
(l+1)
(l)
(3)
r2
= L r2 + ! !A2 AT1
!
(l+1)
0
r1
r1(l)
0
I
!A1 AT2
where
L(3)
!
A1 AT1
= !A2 AT1
0
0
I
!A1 AT2
−1
b1
0
0 b2 ;
0
A1 AT1
−1
0
(1 − !)A1 AT1
0
0
0
A1 AT1
0
(1 − !)I
0
(3.1)
−!I
:
0
T
(1 − !)A1 A1
(3.2)
Hence, we propose the following 3-Block SOR algorithm for solving problem (1.1) as follows.
Algorithm 3.1.
1. Set y(0) ;
2. Calculate r1(0) and r2(0) ;
3. Calculate the iterative parameter !;
4. Iterate for l = 1; 2; : : : ; until “Convergence”
y(l+1) = (1 − !)y(l) + !(A1 AT1 )−1 (b1 − r1(l) );
r2(l+1) = (1 − !)r2(l) + !(b2 − A2 AT1 y(l+1) );
r1(l+1) = (1 − !)r1(l) − !(A1 AT1 )−1 A1 AT2 r2(l+1) :
5
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
3.2. Convergence and optimal factor
Now we shall discuss the convergence of the 3-block SOR method for rank-de cient least-squares
problems. In this case, the Jacobi matrix J3 is
0
J3 = −A2 AT1
0
0
0
−(A1 AT1 )−1 A1 AT2
−(A1 AT1 )−1
:
0
0
(3.3)
Lemma 3.1. The eigenvalues of J3 in (3.3) lie in the real interval
I3 := [− 2=3 ; 0];
(3.4)
where = kA2 AT1 (A1 AT1 )−1 k2 .
Proof. Suppose that is eigenvalue of J and the corresponding eigenvector is (xT ; yT ; z T )T . Then
by the de nition of eigenvalue, there is the following relation:
x
x
J3 y = y ;
z
z
which is
−(A1 AT1 )−1 z = x;
−A2 AT1 x = y;
(3.5)
−(A1 AT1 )−1 A1 AT2 y = z:
It follows from (3.4) that there is
T
= 3 z;
−P Pz
(3.6)
where
P = A2 AT1 (A1 AT1 )−1 :
(3.7)
T
T
− 2 63 60.
Since P P is symmetric and semi-positive de nite, and 3 is eigenvalue of −P P,
Similar to the proof given by Niethammer, et al. [7], we can show the following convergence
result for the 3-block SOR Algorithm 3.1.
Theorem 3.2. The 3-block SOR method of (3.1) for rank-de cient least-squares problem of (1.1)
converges for ! in some interval if and only if
¡ 33=2 ≈ 5:196152;
2.
where = kA2 AT1 (A1 AT1 )−1 k2 = kPk
(3.8)
6
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
In particular; if ¡ 23=2 ≈ 2:828427; then the method of (3.1) converges for all ! such that
0¡!¡
2
;
1 + 2=3
(3.9)
and if 23=2 6 ¡ 33=2 , then the method of (3.1) converges for all ! such that
2
2=3 − 2
¡!¡
;
2=3 − 1
1 + 2=3
(3.10)
and diverges for all other values of !. Furthermore, if ¡ 33=2 , then optimal 3-block SOR relaxation parameter !b(3) is given by
!b(3) =
√
√
3
[( + 1 + 2 )1=3 + ( − 1 + 2 )1=3 ];
(3.11)
is given by
and the spectral radius of the optimum iterative matrix L(3)
!(3)
b
(L(3)
)=2−
!(3)
b
( +
√
1+
2 )2=3
6
√
= 2(1 − !b(3) ):
2
2=3
+ 1 + ( 1 + − )
(3.12)
4. The 2-block SOR method
4.1. The 2-block SOR algorithm
Consider the 2-block diagonal matrix D2 as
A1 AT1
D2 = A2 AT1
0
0
0
I
0
0 A1 AT1
for the system of (2.8). Then the corresponding Jacobi matrix J2 is
0
J2 = 0
0
0
0
T
−P
−(A1 AT1 )−1
;
P
0
(4.1)
where P = A2 AT1 (A1 AT1 )−1 and the 2-block SOR method de ned as follows:
y(l+1)
y(l)
A1 AT1
r2(l+1) = L(2) r2(l) + ! A2 AT1
!
0
r1(l+1)
r1(l)
where
L(2)
!
A1 AT1
= A2 AT1
0
0
I
!A1 AT2
0
I
!A1 AT2
−1
b1
0
0 b2 ;
0
A1 AT1
−1
(1 − !)A1 AT1
0
(1 − !)A2 AT1
0
T
A1 A1
0
0
(1 − !)I
0
(4.2)
−!I
:
0
T
(1 − !)A1 A1
Hence, we propose the following 2-block SOR algorithm for solving the problem of (1.1).
(4.3)
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
7
Algorithm 4.1.
1. Set y(0) ;
2. Calculate r1(0) and r2(0) ;
3. Calculate the iterative parameter !;
4. Iterate for l = 1; 2; : : : ; until “Convergence”
y(l+1) = (1 − !)y(l) + !(A1 AT1 )−1 (b1 − r1(l) );
r2(l+1) = (1 − !)r2(l) + !(b2 − A2 AT1 y(l) ) + A2 AT1 (y(l+1) − y(l) );
r1(l+1) = (1 − !)r1(l) − !(A1 AT1 )−1 A1 AT2 r2(l+1) :
4.2. Convergence and optimal factor
Lemma 4.1. Let be eigenvalue of the Jacobi matrix J2 in (4.1) for the 2-block SOR method.
Then the spectrum of J2 is pure imaginary; that is;
2 60:
Proof. Since
0 (A1 AT1 )−1 P
T
2
J2 = 0
−P P
0
0
the results is true.
0
0 ;
T
−P P
Lemma 4.2. (Young [11]). Let x be any root of the real quadratic equation x2 − bx + c = 0. Then
|x| ¡ 1 if and only if
|c| ¡ 1;
|b| ¡ 1 + c;
where b and c are real.
The convergence result of the 2-block SOR method for the rank de cient problem follows from
these two lemmas and Theorem 2.3.
Theorem 4.3. The 2-block SOR method of (4.2) for the rank-de cient least-squares problem of
(1.1) converges for all ! in the interval
0¡!¡
2
;
1+
(4.4)
2 = kA2 AT1 (A1 AT1 )−1 k2 = (A1 )(A1 ; A2 ), (A1 ) = A(1) =A(k) and (A1 ; A2 ) = A(1) =A(k) ;
where = kPk
1
1
2
1
respectively; are spectral condition number of A1 and relative spectral condition number [12] of A1
8
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
and A2 , A(i) is the ith singular value of A. Furthermore; the optimum SOR relaxation parameter
!b(2) is given by
!b(2) =
2
√
;
1 + 1 + 2
(4.5)
and the spectral radius of the optimum iteration matrix L(2)
is given by
!(2)
b
L(2)
!b(2)
=
√
1 + 1 + 2
2
=
!
(2) 2
!b
2
:
(4.6)
Proof. The proof of the convergence region is the same as the proof of Markham et al. in [5]. We
will omit the proof. Similar to the proof of Markham et al. [5] or of Young [11], we can show
the optimal factor results. Here we want to give the di erent proof on this issue. We de ne the
functions s; t : [0; ] −→ R as
s() =
[2(1 − !) − !2 ]
2
(4.7)
and
[!2 − 4(1 − !)]
:
4
It follows from the eigenvalue functional of (2.9) in Theorem 2.3 that
t() =
q
= s(2 )+− t(2 )!||:
(4.8)
(4.9)
It is evident that s() and t() are an ane decreasing function and an increasing p
function of ,
respectively, and !2√
||t() is quadratic function of . If t ¡ 0, then ||6|s| + !|| |t|; if t ¿ 0,
then ||6|s| + !|| t. It is well-known that (L(2)
! )¿|1 − !|. Then if there exists ! such that
||6|1 − !| for all and all 2 ∈ [0; 2 ], then we obtain the optimal factor !b(2) and spectral radius
(L!(2) ). Since = 0 as = 0, we can get ||6|1 − !| if we restrict s( 2 ) = −(1 − !) and
b
t( 2 ) = 0. In view of the above consideration, the following conditions
!2 2 − 4(1 − !) = 0
(4.10)
(1 − !) − !2 2 =2 = −(1 − !)
(4.11)
and
imply ||6|1 − !|. If the condition of (4.10) holds, then the condition of (4.11) must hold. Then
the optimal factor !b is the unique positive root of Eq. of (4.10). Solving (4.10), we obtain (4.5)
and (4.6).
5. Comparison and conclusion
Analogue of proof given by Markham et al. in [5], we can show the following comparison result.
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
9
) for the 2-block optimum SOR method and
) and (L(3)
Theorem 5.1. The spectral radii (L(2)
!b(3)
!b(2)
the 3-block optimum SOR method de ned in (4.6) and (3.12), respectively, for the rank-de cient
least-squares problem of (1.1) satisfy the following inequality for all ¿ 0:
(L(2)
) ¡ (L(3)
):
!(2)
!(3)
b
(5.1)
b
Analyzing two algorithms, we know that Algorithms 3.1 and 4.1 have the same requirement on
storage and multiplications. In terms of the convergence theory, we know that the 2-block SOR
method always converges for certain interval of !, while the 3-block SOR method does not. It
follows from Theorem 5.1 that the 2-block optimum SOR method converges faster than the 3-block
optimum SOR method. Therefore, the 2-block SOR method is better than the 3-block SOR method
for the rank-de cient problem of (1.1) like the full rank case.
References
[1] M. Benzi, C.D. Meyer, A direct projection method for sparse linear systems, SIAM J. Sci. Comput. 16(1995)
1159–1176.
Bjorck, J.Y. Yuan, Preconditioner for Least Squares Problems by LU Decomposition, Lecture Notes on Scienti c
[2] A.
Computing, Springer, Berlin, 1998, accepted.
[3] R. Freund, A note on two block SOR methods for sparse least-squares problems, LAA, 88/89 (1987) 211–221.
[4] Z. Luo, B.P.B. Silva, J.Y. Yuan, Notes on direct projection method, Technical Report, Department of Mathematics,
UFPR, 1997.
[5] T.L. Markham, M. Neumann, R.J. Plemmons, Convergence of a direct-iterative method for large-scale least-squares
problems, LAA 69(1985) 155–167.
[6] V.A. Miller, M. Neumann: Successive overrelaxation methods for solving the rank de cient least squares problem,
LAA 88/89(1987) 533–557.
[7] W. Niethammer, J. de Pillis, R.S. Varga, Convergence of block iterative methods applied to sparse least squares
problems, LAA 58(1985) 327–341.
[8] C.H. Santos, Iterative methods for rank de cient least squares problems, Master Dissertation, UFPR, Curitiba, Brazil,
1997.
[9] B.P.B. Silva, Topics on numerical linear algebra, Master Dissertation, UFPR, Curitiba, Brazil, 1997.
[10] B.P.B. Silva, J.Y. Yuan, Preconditioner for least squares problems by QR decomposition, Technical Report,
Department of Mathematics, UFPR, Brazil, 1997.
[11] D.M. Young, Iterative Solution of large linear systems, Academic Press New York, 1971.
[12] J.Y. Yuan, R.J.B. de Sampaio, W. Sun, Algebraic relationships between updating and downdating least-squares
problems, Numer. Math. J. Chinese Univ. 3(1996) 203–210 (in Chinese).
Block SOR methods for rank-de cient least-squares problems
C.H. Santos ∗ , B.P.B. Silva, J.Y. Yuan 1
Departamento de Matematica - UFPR, Centro Politecnico, CP: 19.081, CEP: 81531-990, Curitiba, Parana, Brazil
Received 31 August 1997; received in revised form 2 July 1998
Abstract
Many papers have discussed preconditioned block iterative methods for solving full rank least-squares problems. However very few papers studied iterative methods for solving rank-de cient least-squares problems. Miller and Neumann
(1987) proposed the 4-block SOR method for solving the rank-de cient problem. Here a 2-block SOR method and a
3-block SOR method are proposed to solve such problem. The convergence of the block SOR methods is studied. The
optimal parameters are determined. Comparison between the 2-block SOR method and the 3-block SOR method is given
c 1998 Elsevier Science B.V. All rights reserved.
also.
AMS classi cation: 65F10
Keywords: Preconditioned iterative method; Block SOR method; Convergence; Least-squares problem; Rank-de cient
least-squares problem; Optimal parameter
1. Introduction
We frequently meet rank-de cient least-squares problems de ned by
minn kAx − bk2 ;
x∈r
(1.1)
where A is m×n matrix with m¿n and rank (A) ¡ n, when we solve the actual problems in statistics,
economics, genetics, di erential equations, and image and signal processing. Recently, least-squares
methods have got more attention in application areas, and also in applied mathematics society.
Many papers have studied the solvers of the rank-de cient least-squares problems. The general
and ecient ways are singular-value decomposition and QR decomposition. It is well-known that the
iterative methods are preferable for large sparse problems. For rank-de cient least-squares problems,
∗
1
Corresponding author.
Partially supported by grant 301035/93-8 of CNPq, Brazil.
c 1998 Elsevier Science B.V. All rights reserved.
0377-0427/98/$ – see front matter
PII: S 0 3 7 7 - 0 4 2 7 ( 9 8 ) 0 0 1 1 4 - 9
2
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
there are very few papers to study the iterative methods for solving rank-de cient problems. Miller
and Neumann discussed the SOR method to solve the problem (1987) [6]. They partitioned the
matrix A into four parts and then applied the SOR method to solve the new system. In fact, the big
problem for iterative methods to solve the rank-de cient problems is the determination of the rank
of the matrix A. The problem is simple in theory, but not in applications.
Recently, Bjorck and Yuan [2] proposed three algorithms to nd linearly independent rows of
the matrix A by LU factorization, Luo, et al. [4] used the basic solution method (Benzi and Meyer
called direct-projection method [1]) to nd rank of A, and Silva and Yuan [10] applied the QR
decomposition in column-wise to nd the set of linearly independent rows of A. Those methods
motivate us to consider iterative methods for solving the rank-de cient problems. We can nd the
desired preconditioner for rank-de cient least-squares problems by their algorithms.
Since there are many solutions for the rank-de cient problems and we are generally interested in
just the minimum 2-norm solution, we shall derive a new system for the minimum 2-norm solution.
We shall study block SOR methods to solve the new equation by preconditioning technique. Our
block SOR methods are di erent from Miller and Neumann’s SOR method. We shall study the
convergence and optimal relaxation parameter of the block SOR methods, and give some comparison
result between our block SOR methods. Like full rank case, we show the 2-block SOR method always
converges for certain value of the relaxation parameter.
The outline of this paper is as follows. The new system of normal equation for the rank-de cient
least-squares problems is derived in Section 2. The 3-block SOR method is proposed in Section 3.
Its convergence and optimal parameter are also discussed in the section. The convergence theory
shows the 3-block SOR method just converges for some certain case with certain conditions. In
Section 4, the 2-block SOR method is studied. The convergence of the 2-block SOR method tells
us the 2-block SOR method is always convergent for rank-de cient problems with certain relaxation
parameter like the 2-block SOR method for full rank problems. The optimal parameter for the 2block SOR method is given as well in Section 4. The comparison between the 2-block SOR method
and the 3-block SOR method is investigated. The result shows the 2-block SOR method is better
than the 3-block SOR method for rank-de cient problems as full rank problems. Throughout the
paper, we always assume that the matrix A with rank (A) = k ¡ n has the partition
A=
A1
A2
(1.2)
;
where A1 ∈ Rk×n is full row rank, and A2 ∈ R(m−k)×n .
2. New system of normal equation
For the treatment of problem (1.1), we need the following lemma.
Lemma 2.1. Assume that the matrix A has the structure of (1.2). Then
N(A) = N(A1 )
and
R(AT ) = R(AT1 );
where R(A) and N (A) are range and null of A; respectively.
(2.1)
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
3
Proof. ∀x ∈ N (A), it follows from Ax = 0 that A1 x = 0, that is x ∈ N (A1 ). Hence
N (A) ⊂ N (A1 ):
(2.2)
Since A has the structure of (1.2) and rank (A) = k, there exists one nonsingular matrix P such that
PA =
A1
0
(2.3)
:
∀x ∈ N (A1 ), it follows from A1 x = 0 and (2.3) that PAx = 0, that is x ∈ N (A). Thus
N (A1 ) ⊂ N (A):
(2.4)
We have shown N (A) = N (A1 ) by combining (2.2) and (2.4). In terms of N (A) = N (A1 ), it is
obvious that R(AT ) = R(AT1 ).
Since the minimum 2-norm solution x of the rank-de cient least-squares problem of (1.1) is in
R(AT ), that is in R(AT1 ) by Lemma 2.1, We can consider the transformation
x = AT1 y;
(2.5)
where y ∈ Rk , to obtain the minimum 2-norm solution of the problem of (1.1). Substituting (2.5)
into (1.1), we obtain the new system of the normal equation of the problem (1.1) for rank-de cient
case as follows:
A1 AT AAT1 y = A1 AT b:
(2.6)
By the structure of A in (1.2), we can rewrite the system (2.6) as an augmented form
or
AAT1
0
A1 AT1
I
A1 AT
A2 A T
1
0
y
r
0
I
I
0
A1 AT2
A1 AT1
where
r = b − Ax = b −
=
b
0
(2.7)
;
y
b1
r2 = b2 ;
r1
0
AAT1 y
=
r1
r2
;
b=
(2.8)
b1
b2
have the same partition as A in (1.2). Therefore we have shown the following theorem.
Theorem 2.2. Suppose that the matrix A in the problem of (1.1) has the partition of (1.2) with
rank (A) = k = rank (A1 ) ¡ n. Then the minimum 2-norm solution x of the rank-de cient leastsquares problems of (1.1) is given by
x = AT1 y;
where y is the unique solution of the system of (2.7) or (2.8).
4
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
Moreover, the general solution x of the rank-de cient problem of (1.1) is given by
x = AT1 y + z;
∀z ∈ N (A1 ):
The remaining of the paper will discuss block SOR methods to solve the system (2.8). For the
SOR method, there is a well-known eigenvalue functional between the Jacobi iteration matrix and
the SOR iteration matrix.
Theorem 2.3. (Young [11]). Assume that the Jacobi iteration matrix J is weakly p-cyclic and
consistently ordered. Suppose is eigenvalue of J . If satis es the functional
( + ! − 1)p = !p p p−1 ;
(2.9)
then is eigenvalue of the SOR iteration matrix L! . On the contrary, if is eigenvalue of L! ,
and satis es (2.9), then is eigenvalue of J .
3. The 3-block SOR method
3.1. The 3-block SOR algorithm
Consider the 3-block diagonal matrix D = diag (A1 AT1 ; I; A1 AT1 ) for the system of (2.8). We can
obtain the 3-block SOR method as follows:
y(l+1)
y(l)
A1 AT1
(l+1)
(l)
(3)
r2
= L r2 + ! !A2 AT1
!
(l+1)
0
r1
r1(l)
0
I
!A1 AT2
where
L(3)
!
A1 AT1
= !A2 AT1
0
0
I
!A1 AT2
−1
b1
0
0 b2 ;
0
A1 AT1
−1
0
(1 − !)A1 AT1
0
0
0
A1 AT1
0
(1 − !)I
0
(3.1)
−!I
:
0
T
(1 − !)A1 A1
(3.2)
Hence, we propose the following 3-Block SOR algorithm for solving problem (1.1) as follows.
Algorithm 3.1.
1. Set y(0) ;
2. Calculate r1(0) and r2(0) ;
3. Calculate the iterative parameter !;
4. Iterate for l = 1; 2; : : : ; until “Convergence”
y(l+1) = (1 − !)y(l) + !(A1 AT1 )−1 (b1 − r1(l) );
r2(l+1) = (1 − !)r2(l) + !(b2 − A2 AT1 y(l+1) );
r1(l+1) = (1 − !)r1(l) − !(A1 AT1 )−1 A1 AT2 r2(l+1) :
5
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
3.2. Convergence and optimal factor
Now we shall discuss the convergence of the 3-block SOR method for rank-de cient least-squares
problems. In this case, the Jacobi matrix J3 is
0
J3 = −A2 AT1
0
0
0
−(A1 AT1 )−1 A1 AT2
−(A1 AT1 )−1
:
0
0
(3.3)
Lemma 3.1. The eigenvalues of J3 in (3.3) lie in the real interval
I3 := [− 2=3 ; 0];
(3.4)
where = kA2 AT1 (A1 AT1 )−1 k2 .
Proof. Suppose that is eigenvalue of J and the corresponding eigenvector is (xT ; yT ; z T )T . Then
by the de nition of eigenvalue, there is the following relation:
x
x
J3 y = y ;
z
z
which is
−(A1 AT1 )−1 z = x;
−A2 AT1 x = y;
(3.5)
−(A1 AT1 )−1 A1 AT2 y = z:
It follows from (3.4) that there is
T
= 3 z;
−P Pz
(3.6)
where
P = A2 AT1 (A1 AT1 )−1 :
(3.7)
T
T
− 2 63 60.
Since P P is symmetric and semi-positive de nite, and 3 is eigenvalue of −P P,
Similar to the proof given by Niethammer, et al. [7], we can show the following convergence
result for the 3-block SOR Algorithm 3.1.
Theorem 3.2. The 3-block SOR method of (3.1) for rank-de cient least-squares problem of (1.1)
converges for ! in some interval if and only if
¡ 33=2 ≈ 5:196152;
2.
where = kA2 AT1 (A1 AT1 )−1 k2 = kPk
(3.8)
6
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
In particular; if ¡ 23=2 ≈ 2:828427; then the method of (3.1) converges for all ! such that
0¡!¡
2
;
1 + 2=3
(3.9)
and if 23=2 6 ¡ 33=2 , then the method of (3.1) converges for all ! such that
2
2=3 − 2
¡!¡
;
2=3 − 1
1 + 2=3
(3.10)
and diverges for all other values of !. Furthermore, if ¡ 33=2 , then optimal 3-block SOR relaxation parameter !b(3) is given by
!b(3) =
√
√
3
[( + 1 + 2 )1=3 + ( − 1 + 2 )1=3 ];
(3.11)
is given by
and the spectral radius of the optimum iterative matrix L(3)
!(3)
b
(L(3)
)=2−
!(3)
b
( +
√
1+
2 )2=3
6
√
= 2(1 − !b(3) ):
2
2=3
+ 1 + ( 1 + − )
(3.12)
4. The 2-block SOR method
4.1. The 2-block SOR algorithm
Consider the 2-block diagonal matrix D2 as
A1 AT1
D2 = A2 AT1
0
0
0
I
0
0 A1 AT1
for the system of (2.8). Then the corresponding Jacobi matrix J2 is
0
J2 = 0
0
0
0
T
−P
−(A1 AT1 )−1
;
P
0
(4.1)
where P = A2 AT1 (A1 AT1 )−1 and the 2-block SOR method de ned as follows:
y(l+1)
y(l)
A1 AT1
r2(l+1) = L(2) r2(l) + ! A2 AT1
!
0
r1(l+1)
r1(l)
where
L(2)
!
A1 AT1
= A2 AT1
0
0
I
!A1 AT2
0
I
!A1 AT2
−1
b1
0
0 b2 ;
0
A1 AT1
−1
(1 − !)A1 AT1
0
(1 − !)A2 AT1
0
T
A1 A1
0
0
(1 − !)I
0
(4.2)
−!I
:
0
T
(1 − !)A1 A1
Hence, we propose the following 2-block SOR algorithm for solving the problem of (1.1).
(4.3)
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
7
Algorithm 4.1.
1. Set y(0) ;
2. Calculate r1(0) and r2(0) ;
3. Calculate the iterative parameter !;
4. Iterate for l = 1; 2; : : : ; until “Convergence”
y(l+1) = (1 − !)y(l) + !(A1 AT1 )−1 (b1 − r1(l) );
r2(l+1) = (1 − !)r2(l) + !(b2 − A2 AT1 y(l) ) + A2 AT1 (y(l+1) − y(l) );
r1(l+1) = (1 − !)r1(l) − !(A1 AT1 )−1 A1 AT2 r2(l+1) :
4.2. Convergence and optimal factor
Lemma 4.1. Let be eigenvalue of the Jacobi matrix J2 in (4.1) for the 2-block SOR method.
Then the spectrum of J2 is pure imaginary; that is;
2 60:
Proof. Since
0 (A1 AT1 )−1 P
T
2
J2 = 0
−P P
0
0
the results is true.
0
0 ;
T
−P P
Lemma 4.2. (Young [11]). Let x be any root of the real quadratic equation x2 − bx + c = 0. Then
|x| ¡ 1 if and only if
|c| ¡ 1;
|b| ¡ 1 + c;
where b and c are real.
The convergence result of the 2-block SOR method for the rank de cient problem follows from
these two lemmas and Theorem 2.3.
Theorem 4.3. The 2-block SOR method of (4.2) for the rank-de cient least-squares problem of
(1.1) converges for all ! in the interval
0¡!¡
2
;
1+
(4.4)
2 = kA2 AT1 (A1 AT1 )−1 k2 = (A1 )(A1 ; A2 ), (A1 ) = A(1) =A(k) and (A1 ; A2 ) = A(1) =A(k) ;
where = kPk
1
1
2
1
respectively; are spectral condition number of A1 and relative spectral condition number [12] of A1
8
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
and A2 , A(i) is the ith singular value of A. Furthermore; the optimum SOR relaxation parameter
!b(2) is given by
!b(2) =
2
√
;
1 + 1 + 2
(4.5)
and the spectral radius of the optimum iteration matrix L(2)
is given by
!(2)
b
L(2)
!b(2)
=
√
1 + 1 + 2
2
=
!
(2) 2
!b
2
:
(4.6)
Proof. The proof of the convergence region is the same as the proof of Markham et al. in [5]. We
will omit the proof. Similar to the proof of Markham et al. [5] or of Young [11], we can show
the optimal factor results. Here we want to give the di erent proof on this issue. We de ne the
functions s; t : [0; ] −→ R as
s() =
[2(1 − !) − !2 ]
2
(4.7)
and
[!2 − 4(1 − !)]
:
4
It follows from the eigenvalue functional of (2.9) in Theorem 2.3 that
t() =
q
= s(2 )+− t(2 )!||:
(4.8)
(4.9)
It is evident that s() and t() are an ane decreasing function and an increasing p
function of ,
respectively, and !2√
||t() is quadratic function of . If t ¡ 0, then ||6|s| + !|| |t|; if t ¿ 0,
then ||6|s| + !|| t. It is well-known that (L(2)
! )¿|1 − !|. Then if there exists ! such that
||6|1 − !| for all and all 2 ∈ [0; 2 ], then we obtain the optimal factor !b(2) and spectral radius
(L!(2) ). Since = 0 as = 0, we can get ||6|1 − !| if we restrict s( 2 ) = −(1 − !) and
b
t( 2 ) = 0. In view of the above consideration, the following conditions
!2 2 − 4(1 − !) = 0
(4.10)
(1 − !) − !2 2 =2 = −(1 − !)
(4.11)
and
imply ||6|1 − !|. If the condition of (4.10) holds, then the condition of (4.11) must hold. Then
the optimal factor !b is the unique positive root of Eq. of (4.10). Solving (4.10), we obtain (4.5)
and (4.6).
5. Comparison and conclusion
Analogue of proof given by Markham et al. in [5], we can show the following comparison result.
C.H. Santos et al. / Journal of Computational and Applied Mathematics 100 (1998) 1–9
9
) for the 2-block optimum SOR method and
) and (L(3)
Theorem 5.1. The spectral radii (L(2)
!b(3)
!b(2)
the 3-block optimum SOR method de ned in (4.6) and (3.12), respectively, for the rank-de cient
least-squares problem of (1.1) satisfy the following inequality for all ¿ 0:
(L(2)
) ¡ (L(3)
):
!(2)
!(3)
b
(5.1)
b
Analyzing two algorithms, we know that Algorithms 3.1 and 4.1 have the same requirement on
storage and multiplications. In terms of the convergence theory, we know that the 2-block SOR
method always converges for certain interval of !, while the 3-block SOR method does not. It
follows from Theorem 5.1 that the 2-block optimum SOR method converges faster than the 3-block
optimum SOR method. Therefore, the 2-block SOR method is better than the 3-block SOR method
for the rank-de cient problem of (1.1) like the full rank case.
References
[1] M. Benzi, C.D. Meyer, A direct projection method for sparse linear systems, SIAM J. Sci. Comput. 16(1995)
1159–1176.
Bjorck, J.Y. Yuan, Preconditioner for Least Squares Problems by LU Decomposition, Lecture Notes on Scienti c
[2] A.
Computing, Springer, Berlin, 1998, accepted.
[3] R. Freund, A note on two block SOR methods for sparse least-squares problems, LAA, 88/89 (1987) 211–221.
[4] Z. Luo, B.P.B. Silva, J.Y. Yuan, Notes on direct projection method, Technical Report, Department of Mathematics,
UFPR, 1997.
[5] T.L. Markham, M. Neumann, R.J. Plemmons, Convergence of a direct-iterative method for large-scale least-squares
problems, LAA 69(1985) 155–167.
[6] V.A. Miller, M. Neumann: Successive overrelaxation methods for solving the rank de cient least squares problem,
LAA 88/89(1987) 533–557.
[7] W. Niethammer, J. de Pillis, R.S. Varga, Convergence of block iterative methods applied to sparse least squares
problems, LAA 58(1985) 327–341.
[8] C.H. Santos, Iterative methods for rank de cient least squares problems, Master Dissertation, UFPR, Curitiba, Brazil,
1997.
[9] B.P.B. Silva, Topics on numerical linear algebra, Master Dissertation, UFPR, Curitiba, Brazil, 1997.
[10] B.P.B. Silva, J.Y. Yuan, Preconditioner for least squares problems by QR decomposition, Technical Report,
Department of Mathematics, UFPR, Brazil, 1997.
[11] D.M. Young, Iterative Solution of large linear systems, Academic Press New York, 1971.
[12] J.Y. Yuan, R.J.B. de Sampaio, W. Sun, Algebraic relationships between updating and downdating least-squares
problems, Numer. Math. J. Chinese Univ. 3(1996) 203–210 (in Chinese).