Directory UMM :Data Elmu:jurnal:J-a:Journal of Computational And Applied Mathematics:Vol103.Issue2:
Journal of Computational and Applied Mathematics 103 (1999) 307–321
Convergence of partially asynchronous block
quasi-Newton methods for nonlinear systems of equations
Jian-Jun Xu ∗
Department of Mathematics, Temple University, Philadelphia, PA 19122, United States
Received 4 May 1998; received in revised form 7 October 1998
Abstract
In this paper, a partially asynchronous block Broyden method is presented for solving nonlinear systems of equations of
the form F(x) = 0. Sucient conditions that guarantee its local convergence are given. In particular, local convergence is
shown when the Jacobian F ′ (x∗ ) is an H -matrix, where x∗ is the zero point of F. The results are extended to Schubert’s
c 1999 Elsevier Science B.V. All rights
method. A connection with discrete Schwarz alternating procedure is also shown.
reserved.
Keywords: Parallel iterative methods; Asynchronous iterations; Quasi-Newton methods; Schwarz alternating procedure
1. Introduction
Chazan and Miranker rst introduced an asynchronous method for the solution of nonsingular
linear system of equations of the form Ax = b [9]. They proved that the asynchronous point method
converges to the solution if and only if A is an H -matrix [29]. This idea was extended to nonlinear
systems of equations, and there is now a considerable understanding on the convergence properties
of asynchronous iterative methods for nonlinear problems. Many authors have concentrated on xed
point problems of the form x = G(x). Convergence properties have been shown if G satis es some
contraction properties; see, e.g., [3–5, 13]. The xed point mapping G does not always arise directly
from the formulation of physical problems, however, and when it does, the contracting properties
are not always easily veri able.
In this paper, we consider the problem of nding x∗ which satis es the following nonlinear system
of equations:
F(x) = 0;
(1.1)
T
n
where F = (f1 ; : : : ; fn ) is a nonlinear operator from R into itself.
∗
E-mail: [email protected].
c 1999 Elsevier Science B.V. All rights reserved.
0377-0427/99/$ – see front matter
PII: S 0 3 7 7 - 0 4 2 7 ( 9 8 ) 0 0 2 6 8 - 4
308
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
In the literature, one can nd two general approaches for the asynchronous solutions of (1.1).
First, for an asynchronous implementation of Newton’s method, one can use two processes, one
updating the iterates, and the other evaluating the Jacobian matrix; see [3, 7, 20]. Also see [31] for
the analogue of Broyden’s method. More recently, asynchronous block methods, or more general
multisplitting methods were proposed. In [2, 6] the nonlinear term was treated explicitly because of
the weak nonlinearity. In [14, 22], convergence properties were shown where F is an M -function,
see also [12, 15] for point methods. In [12, 14, 15, 22], exact solutions for subproblems were
assumed. The use of Newton-type methods for approximate solutions of the subproblems and local
convergence theorems for the corresponding asynchronous methods were given in [1, 27, 32].
In this paper, we consider partially asynchronous block quasi-Newton methods.
Suppose F and x in (1.1) are conformally partitioned as follows:
F = (F1T ; : : : ; FLT )T ;
x = (x1T ; : : : ; xLT )T ;
where Fi : R n → Rni ; Fi = (fi1 ; : : : ; fini )T ; xi ∈ Rni ; xi = (xi1 ; : : : ; xini )T ; i = 1; : : : ; L: Let Si = {i1 ; : : : ; ini },
SL
the partition chosen is such that i=1
Si = {1; : : : ; n}; Si ∩ Sj = ∅; i 6= j; i; j = 1; : : : ; L. This partition
may correspond to a decomposition of the domain of a nonlinear partial di erential equations, see
Section 5.
The system (1.1) can be rewritten as
Fl (x1 ; : : : ; xl ; : : : ; xL ) = 0;
l = 1; : : : ; L:
(1.1)′
We consider the following nonlinear block method. Given initial values of x = (x1T ; : : : ; xLT )T , repeat
the following procedure until convergence:
For l = 1; : : : ; L
Solve for yl in Fl (x1 ; : : : ; xl−1 ; yl ; xl+1 ; : : : ; xL ) = 0;
for all l ∈ S ⊆{1; : : : ; L};
Set xl = yl for all l ∈ S:
(1.2)
In (1.2) the subset S can be chosen in a dynamic fashion. The classical nonlinear block-Jacobi
method and nonlinear block-Gauss–Seidel method [4, 23] are two examples of such selections. For
the purpose of parallel processing, the nonlinear block Jacobi method is nearly ideal, since up
to L processors can each perform one of the iterations in (1.2) with the others simultaneously.
It is synchronous in the sense that to begin the computation of the next iterate, each processor has
to wait until all processors have completed their current iteration. In practice we need to add a
synchronization mechanism to ensure that the algorithm is carried out correctly. The time necessary
to carry out the synchronization mechanism, as well as the time a processor must wait until its
data is ready to continue the calculation, adds an overhead to the computation. By removing this
synchronization and letting the processors continue their calculations according to the information
currently available, we obtain the so-called asynchronous parallel methods; see, e.g., [3, 4, 9].
Generally, the exact solutions for the subproblems are not available. Approximate methods such as
Newton’s method, Broyden’s method etc., (see, e.g., [11, 23]) can be considered to get approximate
solutions. The use of Newton-type methods were given in [1, 27, 32].
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
309
The advantages of quasi-Newton methods for (1.1) are that they involve evaluating only O(n)
scalar functions in each iteration step, which is much cheaper than Newton’s method, they do not
involve computation of the derivative of F, and they still have superlinear convergence; see, e.g. [10,
11] and the references given therein. In this paper, we consider the use of quasi-Newton methods
instead of Newton’s method. We will consider two quasi-Newton methods: Broyden’s method [8]
and Schubert’s method [24].
The main purpose of this paper is to give sucient conditions which guarantee local convergence
of partially asynchronous block quasi-Newton methods. To this end, we assume in the rest of the
paper that F and the partition satisfy the following conditions:
F is continuously di erentiable on an open convex set
⊂ R n :
(1.3)
There exists a constant ¿ 0 such that
kF ′ (x) − F ′ (x∗ )k6 kx − x∗ k;
for x ∈
:
(1.4)
All the matrices @Fi (x∗ )=@xi ; i = 1; : : : ; L; are nonsingular, moreover
(|D(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))|) ¡ 1;
(1.5)
where D(x∗ ) = diag(@F1 (x∗ )=@x1 ; : : : ; @FL (x∗ )=@xL ) is a block diagonal matrix, and we use the notation
@fi1 (x)
@xj
1
@Fi (x)
= ···
@f (x)
@xj
ini
@xj1
···
···
···
@fi1 (x)
@xjnj
···
@fini (x)
@xjnj
i; j = 1; : : : ; L;
and
@F1 (x)
@x
1
F ′ (x) =
···
@FL (x)
@x1
···
···
···
@F1 (x)
@xL
···
:
@FL (x)
@xL
Remark 1. Conditions (1.3) and (1.4) are standard. Condition (1.5) is natural for the convergence.
Consider the linear case, i.e., F(x) = Ax − b, and let F and x be partitioned as before. Suppose that
A−1 exists. Since F ′ (x) = A, condition (1.5) is necessary and sucient for the convergence of the
asynchronous block method for the linear equations F(x) = 0; see, e.g., [4, 9].
Remark 2. Condition (1.5) holds when the Jacobian matrix F ′ (x∗ ) is an H -matrix, since F ′ (x∗ ) =
D(x∗ ) − (D(x∗ ) − F ′ (x∗ )) is an H -splitting of F ′ (x∗ ); see e.g., [5, 15]. On the other hand, if each
block has only one component, i.e., if nl = 1; l = 1; : : : ; L, then (1.5) is equivalent to F ′ (x∗ ) being an
310
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
H -matrix. The class of H -matrices includes M -matrices, and also the class of strictly or irreducibly
diagonally dominant matrices [29].
In Section 2, we present both the computational and mathematical models of partially asynchronous
block Broyden’s method. The local convergence theorem is proved in Section 3 under conditions
(1.3)–(1.5). In Section 4, we point out that local convergence is still guaranteed if Schubert’s method
[24] is used instead of Broyden’s method. The connection of the block method with the Schwarz
alternating procedure (see, e.g., [18, 19, 25, 30]) is discussed in Section 5.
The weighted maximum norms is used as an important tool in proofs of asynchronous iterative
methods; see, e.g., [3, 4, 6, 9, 13, 17, 26, 27]. Given an vector w ∈ R n ; w ¿ 0. Let w = (w1T ; : : : ; wLT )T
be partitioned conformally with x, then we de ne
)
(
|xij |
kxkw = max{kxi kwi ; 16i 6L} = max
; 16j 6ni ; 16i6L :
wij
Given a matrix A ∈ R n×n . Let A = (Aij )L × L be partitioned conformally with F ′ (x), then the norms
k · kw ; k · kwi induce the following matrix norm:
kAxkw
kAkw = max
: x 6= 0; x ∈ R n :
kxkw
We extend the weighted maximum norms for rectangular matrices as follows:
k(Ai1 ; : : : ; AiL )xkwi
: x 6= 0; x ∈ R n
k(Ai1 ; : : : ; AiL )kwi = max
kxkw
and
(
kAij xj kwi
kAij kwi = max
: xj 6= 0; xj ∈ R nj
kxj kwj
)
:
We immediately have
kAii (Ai1 ; : : : ; AiL )kwi 6kAii kwi · k(Ai1 ; : : : ; AiL )kwi ;
(1.6)
k|A|kw = kAkw ;
(1.7)
k|(Ai1 ; : : : ; AiL )|kwi = k(Ai1 ; : : : ; AiL )kwi ;
kAkw = max{k(Ai1 ; : : : ; AiL )kwi ; 16i 6L};
(1.8)
kAij kwi 6k(Ai1 ; : : : ; AiL )kwi ;
(1.9)
and
16i; j 6L:
By the theory of nonnegative matrix (see, e.g., [4, 28]), condition (1.5) is equivalent to
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
311
(1:5)′ All matrices @Fi (x∗ )=@xi ; i = 1; : : : ; L; are nonsingular, moreover, there exists 0 ¡ 1, and a
vector w ¿ 0 such that
kD(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))kw ¡ 0 :
2. Partially asynchronous block Broyden method
Broyden’s method [8] for (1.1) is as follows:
x(k + 1) = x(k) − B(k)−1 F(x(k));
B(k + 1) = B(k) + (1=z(k)T z(k)) · (y(k) − B(k)z(k))z(k)T ;
where y(k) = F(x(k + 1)) − F(x(k));
k = 1; 2; : : : ;
z(k) = x(k + 1) − x(k);
where x(0) and B(0) are initial approximation to the solution x∗ and initial approximation the
Jacobian F ′ (x∗ ), respectively.
We will rst describe a computational model for partially asynchronous block Broyden’s method,
i.e., how the computer is actually programmed to execute its instructions, then we give a mathematics
model, i.e., how the execution is described mathematically in order to analyze convergence [26].
Let us as before assume that there are L processors. We design L processes. Each processor
perform computation in one process. In each process, say the lth process, we use one step of
Broyden’s method to update the lth block xl in the lth subproblem in (1.2), l = 1; : : : ; L. More
precisely, the computational model can be written as the following pseudo-code:
Given an initial guess x, and initial matrices Bl ; l = 1; : : : ; L.
Process l(l = 1; : : : ; L)
Initialization for x; Bl
Step 1: Compute Fl := Fl (x);
Step 2: Set x := xl ; Fl := Fl ;
Step 3: Compute xl := xl − Bl−1 Fl ;
Step 4: Send xl to the other processes;
Step 5: Check the termination criterion, if satis ed then stop;
Step 6: Receive the values of x (except xl ) from the other processes;
Step 7: Compute Fl := Fl (x);
Step 8: Compute z := xl − x l ; y := Fl − Fl ;
Step 9: Compute Bl := Bl + (1=z T z) · (y − Bl z)z T ;
Step 10: Goto step 2;
where Bl ; F l ; x;
y; z are local variables, only x is a global variable. In each cycle of a process, say
the lth process, one step of Broyden’s update is performed for a subproblem of nl dimension. The
L processes can be done in parallel. There is no explicit synchronization among the processes.
Remark 3. Since Broyden update is a rank-one update, the Sherman–Morrison formula can be used
to update the inverse of Bl . Let Hl = Bl−1 , then initialization for Bl is changed for Hl , Steps 3 and 9
are rewritten as follows while the other steps remain unchanged:
Step 3′ : Compute xl := xl − Hl · Fl ;
Step 9′ : Hl := Hl + (1=z T Hl y) · (y − Bl z)z T ;
312
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
One can also uses QR factorization technique to solve the linear system in step 3; see, e.g., [11].
We give the following mathematical model in order to analyze the convergence.
Let the initial approximations x(0) = (x1 (0)T ; : : : ; xL (0)T )T , and Bi (0); i = 1; : : : ; L be given, then for
k = 0; 1; : : : ; i = 1; : : : ; L;
−1
xi (k + 1) = xi (si (k)) − Bi (k) Fi (u(k)) if i ∈ I (k)
else
xi (k + 1) = xi (k)
Bi (k + 1) = Bi (k) + (1=zi (k)T zi (k)) · (yi (k) − Bi (k)zi (k))zi (k)T
Bi (k + 1) = Bi (k)
if i ∈ I (k)
else;
where u(k) = (x1 (s1 (k))T ; : : : ; xL (sL (k))T )T ;
yi (k) = Fi (x(k + 1)) − Fi (v(k; i));
v(k; i) = (x1 (k + 1)T ; : : : ; xi−1 (k + 1)T ;
xi (si (k))T ; xi+1 (k + 1)T ; : : : ; xL (k + 1)T )T ;
zi (k) = xi (k + 1) − xi (si (k));
where I (k) and sj (k) satisfy the following conditions
(2.1) for some xed integer p ¿ 0; k − p ¡ sj (k)6k; j = 1; : : : ; L; k = 0; 1; : : :
(2.2) each I (k); k = 0; 1; : : : are subset of {1; : : : ; L}, and the set {k: j ∈ I (k)} is unbounded for
all j = 1; : : : ; L.
Conditions (2.1) and (2.2) may be interpreted as follows: At each instant of time k, all the ith
block component xi ; i ∈ I (k), are updated while the remaining are unchanged. The updating uses the
rst block component of x(s1 (k)), the second block component of x(s2 (k)), etc. Then the matrices
Bi ; i ∈ I (k), are updated by using the new results. Each pair of block component and matrix is
updated in nitely many often, and no update uses a value of a component which was produced by
an update p or more steps previously.
The assumption of the existence of uniform bound p for delays appeared in the rst asynchronous
model in [9]. It is not an additional constraint in most practical implementations [3]. This type of
asynchronous model was called partially asynchronous method in [4].
We will refer the above mathematical model as partially asynchronous block Broyden’s (PABB)
method.
If we take si (k) = k; I (k) = {1; : : : ; L} for all k and i, then PABB method describes block-Jacobi
Broyden method.
If we take si (k) = k; I (k) = {k + 1 (mod L)} for all k, then PABB method describes block-Gauss–
Seidel Broyden method.
3. Local convergence theorem
In this section we prove the local convergence of the PABB method. We have:
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
313
Theorem 3.1. Suppose that (1:3)–(1:5) are satis ed; then there exist two constants ¿ 0; ¿ 0;
such that if kx(0) − x∗ kw 6 for w ¿ 0 de ned in (1:5)′ ; kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L;
then the sequence {x(k)} generated by PABB method converges to x∗ .
Before we give the proof of Theorem 3.1, we need several Lemmas. Because of the norm equivalence, we may assume that the norm in (1.4) is k · kw , and there exist two constants 1 ; 2 ¿ 0
such that
kAk2 61 kAkwˆ ;
kAkwˆ 62 kAk2 ;
(3.1)
where wˆ = w if A ∈ Rn × n , and wˆ = wi if A ∈ Rni × n (or Rni × nj ).
Lemma 3.2 (Ortega and Rheinboldt [23]). If F : R n → Rm is Gateaux-di erentiable in an open convex set
0 ; then
F(y) − F(x) =
Z
1
F ′ (x + t(y − x))(y − x) dt
for x; y ∈
0 :
0
Lemma 3.3 (Ortega and Rheinboldt [23]). Let A; C ∈ Rn × n ; A is nonsingular and kA−1 k2 6 1 ; kA−
Ck2 6 2 ; 1 2 ¡ 1; then C is also nonsingular; moreover;
kC −1 k2 6 1 =(1 − 1 2 ):
Lemma 3.4. Under the conditions (1:3)–(1:5); we have
@F (x)
@Fi (x∗ )
i
−
6 kx − x∗ kw
@xi
@xi
for x ∈
(3.2)
wi
and there exists S(x∗ ; 1 ) = {x: kx − x∗ k61 } ⊂
; such that
!−1
@Fi (x∗ )
@Fi (x)
@Fi (x) @Fi (x)
@Fi (x)
¡ 0 ;
;:::;
; 0;
;:::;
@xi
@x1
@xi−1
@xi+1
@xL
i = 1; : : : ; L; x ∈ S(x∗ ; 1 );
wi
xi − x∗ −
i
@Fi (x∗ )
Fi (x)
@xi
wi
!
∗ ) −1
@F
(x
1
i
· kx − x∗ k2 ;
6kx − x∗ kw + ·
w
2
@xi
(3.3)
!−1
i = 1; : : : ; L; x ∈ S(x∗ ; 1 ):
(3.4)
wi
Proof. By the properties of the norms k · kw ; k · kwi in Section 1 and (1.4), we immediately have
(3.2). Since kD(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))kw ¡ 0 , and F ′ (x) is continuously di erentiable at x∗ , there
314
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
exists S(x∗ ; 1 ) = {x: kx − x∗ kw 61 } ⊂
; such that
kD(x∗ )−1 (F ′ (x) − D(x))kw ¡ 0 ; for x ∈ S(x∗ ; 1 )
thus by (1.8), (3.3) holds.
By Lemma 3.2, (3.2) and (3.3), we have
@Fi (x∗ )
@xi
kxi − xi∗ −
!−1
Fi (x)kwi
!−1
!
∗)
@Fi (x∗ )
@F
(x
i
∗
∗
Fi (x) − Fi (x ) −
(xi − xi )
=
@xi
@xi
wi
!−1 "Z
1
@Fi (x∗ )
@Fi (x∗ + t(x − x∗ ))
@Fi (x∗ + t(x − x∗ ))
=
;:::;
; 0;
@xi
@x1
@xi−1
0
!
@Fi (x∗ + t(x − x∗ ))
@Fi (x∗ + t(x − x∗ ))
(x − x∗ ) dt
;:::;
@xi+1
@xL
+
Z
1
@F (x∗ + t(x − x∗ ))
i
0
@xi
−
@F (x∗ )
i
@xi
!
#
∗
(xi − xi ) dt
wi
!
∗ ) −1
1
@F
(x
i
· kx − x∗ k2
60 kx − x∗ kw +
w
2
@xi
for x ∈ S(x∗ ; 1 ):
wi
Lemma 3.5 (Dennis and Robert [11]). Let u; v ∈ R n ; uT v = 1; then
kI − uvT k2 = kuk2 · kvk2 :
Lemma 3.6. Let i ∈ I (k); then
@Fi (x∗ )
@Fi (x∗ )
Bi (k + 1) −
6
Bi (k) −
+ 1 (x(k + 1); v(k; i));
@xi
2
@xi
2
where (x(k + 1); v(k; i)) = max{kx(k + 1) − x∗ k2 ; kv(k; i) − x∗ k2 }; 1 = 1 2 .
Proof. If i ∈ I (k), from Lemma 3.5 we have
@Fi (x∗ )
Bi (k + 1) −
@xi
2
!
∗
(yi − @Fi (x ) zi (k))zi (k)T
zi (k)zi (k)T
@Fi (x∗ )
@xi
I−
6
Bi (k) −
+
@xi
zi (k)T zi (k)
2
zi (k)T zi (k)
2
∗
∗
@Fi (x )
1
@Fi (x )
zi (k)
;
6
Bi (k) −
+
yi (k) −
@xi
kzi (k)k2
@xi
2
2
315
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
while by (3.1), (3.2)
@Fi (x∗ )
zi (k)
yi (k) −
@xi
2
@Fi (x∗ )
(xi (k + 1) − xi (si (k)))
=
Fi (x(k + 1)) − Fi (v(k; i)) −
@xi
2
Z
1 @F (v(k; i) + t(x(k + 1) − v(k; i)))
∗
@Fi (x )
i
−
)(xi (k + 1) − xi (si (k))) dt
=
(
0
@xi
@xi
2
6 1 max{kx(k + 1) − x∗ k2 ; kv(k; i) − x∗ k2 }kxi (k + 1) − xi (si (k))k2 ;
thus we have proved Lemma 3.6.
Proof of Theorem 3.1. Suppose that for some xed ¿ 0; we have
!−1
@Fi (x∗ )
6 ;
@xi
kF ′ (x∗ )kw 6 ;
i = 1; : : : ; L:
(3.5)
wi
For any 2 ¿ 0, by Lemma 3.3, there exists ¿ 0 such that if kBi − (@Fi (x∗ ))=@xi k2 62, then Bi
is nonsingular and
−1
B −
i
@F (x∗ )
i
@xi
!−1
62 ;
i = 1; : : : ; L:
(3.6)
2
Let kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L; kx(0) − x∗ kw 6; 61 . We can choose 2 and small
enough, such that
1
1
r ≡ 2 2 ( + 2 ) + 2 + 0 ¡ 1;
2p
1 1 6;
(3.7)
1−r
We use mathematical induction to show the following inequalities:
kxi (k + 1) − xi∗ kwi 6rkx(s(k)) − x∗ kw if i ∈ I (k);
kx (k + 1) − x∗ k 6 kx(k) − x∗ k else;
i
i
wi
w
@Fi (x∗ )
Bi (k + 1) −
62
@xi
(3.8)
2
and as a byproduct we get
kx(k) − x∗ kw 6r q−1 kx(0) − x∗ kw ;
if k ¿ (q − 1) · 2p:
(3.9)
316
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
If i ∈ I (0), by (3.1), (3.4)–(3.6) we have
kxi (1) − xi∗ kwi = kxi (si (0)) − xi∗ − Bi (0)−1 Fi (u(0))kwi
@F (x∗ ) −1
i
−1
− Bi (0)
· kFi (u(0))kwi
6
@xi
w
i
∗
+
xi (si (0)) − xi −
@Fi (x∗ )
@xi
!−1
Fi (u(0))
wi
1
6 2 2 kFi (u(0))kwi + 0 ku(0) − x∗ kw + ku(0) − x∗ k2w :
2
By Lemma 3.2,
kFi (u(0))kwi = kFi (u(0)) − Fi (x∗ )kwi
6 kF(u(0)) − F(x∗ ) − F ′ (x∗ )(u(0) − x∗ )kw + kF ′ (x∗ )(u(0) − x∗ )kw
1
6 ku(0) − x∗ k2w + ku(0) − x∗ kw :
2
Since u(0) = x(0), thus
1
1
kxi (1) − xi∗ kwi 6 2 2
+ + 0 + kx(0) − x∗ kw
2
2
∗
= rkx(0) − x k :
w
If i ∈= I (0), then
kxi (1) − xi∗ kwi = kxi (0) − xi∗ kwi 6kx(0) − x∗ kw :
On the other hand, if i ∈ I (0), by Lemma 3.6
@Fi (x∗ )
@Fi (x∗ )
Bi (1) −
+ 1 (x(1); v(0; i))
6
Bi (0) −
@xi
@xi
2
2
6 + 1 1 kx(0) − x∗ kw 6 + 1 1 62:
If i ∈= I (0),
@Fi (x∗ )
@Fi (x∗ )
Bi (1) −
=
Bi (0) −
62:
@xi
@xi
2
2
Assume that (3.8) holds for k = 0; 1; : : : ; m − 1. It is clear that kx(k) − x∗ kw 6; k = 0; 1; : : : ; m; Bi (k)
is nonsingular, moreover, by (3.6), k(Bi (k))−1 − (@Fi (x∗ ))=@xi )−1 k2 62 ; i = 1; : : : ; L; k = 0; 1; : : : ; m
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
317
Thus if i ∈ I (m),
kxi (m + 1) − xi∗ kwi = kxi (si (m)) − xi∗ − Bi (m)−1 Fi (u(m))kwi
!
∗ ) −1
@F
(x
i
−1
· kFi (u(m))kw
6
i
Bi (m) −
@xi
wi
!
∗ ) −1
@F
(x
i
∗
+
xi (si (m)) − xi −
Fi (u(m))
@xi
wi
1
1
+ ku(m) − x∗ kw + 0 + ku(m) − x∗ kw
2
2
∗
∗
= rku(m) − x kw 6rkx(s(m)) − x kw ;
6 2 2
where s(m) is an element in {si (m); i = 1; : : : ; L}, thus m − p ¡ s(m)6m.
If i ∈= I (m), we have kxi (m + 1) − xi∗ kwi = kxi (m) − xi∗ kwi 6kx(m) − x∗ kw .
Assume that for some h; (h − 1) · 2p ¡ m6h · 2p.
We will show that (3.9) holds for if (q − 1) · 2p ¡ k 6m + 1; 16q6h.
We have already shown (3.9) for q = 1. Assume that it holds for 16q6 q ¡ h. Because of
the uniform bound p for the delays, each block component is updated at least once for every 2p
consecutive iterations, say (k + 1)th; : : : ; (k + 2p)th iterations, and the values of the block components
used for the updating all come from these 2p iterates. Thus for any k; q · 2p ¡ k 6m + 1, and
any i ∈ {1; : : : ; L}, there exists k1 6k − 1; u(k1 ) = (x1 (s1 (k1 ))T ; : : : ; xL (sL (k1 ))T )T ; si (k1 ) ¿ (q − 1) ·
2p; i = 1; : : : ; L; such that
kxi (k) − xi∗ kwi = kxi (k1 + 1) − xi∗ kwi
= kxi (si (k1 )) − xi∗ − Bi (k1 )−1 Fi (u(k1 ))kwi 6rku(k1 ) − x∗ kw
6r · r (q−2)
kx(0) − x∗ k = r (q−1)
kx(0) − x∗ k :
w
w
Thus kx(k) − x∗ kw 6r (q−1)
kx(0) − x∗ kw :
Therefore (3.9) holds for k 6m + 1. Thus if i ∈ I (m); by Lemma 3.6,
@Fi (x∗ )
@Fi (x∗ )
6
Bi (m) −
+ 1 (x(m + 1); v(m; i))
Bi (m + 1) −
@xi
2
@xi
2
@Fi (x∗ )
6
Bi (m) −
+ 1 1 r h−3
@xi
2
@Fi (x∗ )
6 · · · 6
Bi (m − 2p + 1) −
+ 2p1 1 r h−3
@xi
2
@Fi (x∗ )
6 · · · 6
Bi (0) −
+ 2p1 1 (r h−3 + · · · + r + 1)
@xi
2
2p1
6+
62:
1−r
318
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
If i ∈= I (m);
@Fi (x∗ )
@Fi (x∗ )
=
Bi (m) −
62:
Bi (m + 1) −
@xi
@xi
2
2
Therefore (3.8) holds for all k; and then (3.9) holds for all k; thus {x(k)} convergences to x∗ .
By Remark 2, we have the following corollary.
Corollary 3.7. Suppose that (1.3)–(1.4) are satis ed; and F ′ (x∗ ) is an H -matrix; then for any
partitioning of F and x; there exist two constants ¿ 0; ¿ 0; such that if kx(0) − x∗ kw 6 for
w ¿ 0 de ned in (1.5)′ ; kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L; the sequence {x(k)} generated by
PABB method converges to x∗ .
4. Extension to Schubert’s method
Schubert’s method [24] is a variation of Broyden’s method which is of interest in the case that
the Jacobian is sparse. In this variation, the approximate Jacobian is forced to have the same sparsity
pattern as the Jacobian.
Given a vector function F = (f1 ; : : : ; fm )T ∈ Rm → Rm ; we de ne its sparsity pattern as a matrix
PF = (pij )m×m ; where
pij =
0 if fi (x) is independent of xj
1 else
i; j = 1; : : : ; m:
Let PFl = (plij ) be the sparsity pattern of Fl ; l = 1; : : : ; L. Using Schubert’s method instead of Broyden’s method for each subproblem in (1.2) leads to the following computational model:
Given initial solution x = (x1T ; : : : ; xLT )T ; and matrices Bl .
Process l(l = 1; : : : L)
Initialization for x = (x1T ; : : : ; xlT ; : : : ; xLT )T ; Bl
Steps 1–8 and 10 are the same as the computational model of PABB method;
step 9: if plij = 1 then uij ←zj else uij ←0; i; j = 1; : : : ; nl ;
Bl ← Bl +
nl
X
i=1
ui 6=0
ei eiT
(y − Bl z) T
ui ;
uiT ui
where ui = (ui1 ; : : : ; uinl )T ; i = 1; : : : ; nl ; and ei ∈ Rnl is the unit vector of which the ith component is
1, i = 1; : : : ; nl .
Its mathematical model can be described similarly as PABB method.
Also its local convergence can be shown in an analogous way to PABB method, since we can
easily get an inequality for Schubert’s method similar to Lemma 3.6, see [21] ((3.14), p. 593).
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
319
5. Connection with the discrete Schwarz alternating procedure
In this section, suppose that there are overlaps among the blocks x1 ; : : : ; xL ; and correspondingly
among F1 ; : : : ; FL . This may happen when discrete Schwarz alternating procedure (see, e.g., [18, 19,
25, 30]) is used to solve (1.1), where each block xl corresponds to the unknowns in a subdomain.
In this case, the computational models need not change, but we need to modify the mathematical
model so that thePprevious analysis can be applied.
Suppose that Ll= 1 nl = m ¿ n; we de ne a vector x = (x T1 ; : : : ; x TL )T ∈ Rm from x ∈ R n by setting
x l = xl ; l = 1; : : : ; L: Denote this mapping from R n to Rm by x = (x), similarly we obtain F =
(F T1 ; : : : ; F TL )T = (F). We can thus rewrite (1.1) by repeating the overlapping unknowns and the
corresponding equations:
Fl (x l ; Zl ) = 0;
l = 1; : : : ; L;
(5.1)
where the vector Zl is a subset of i6=l {x l }; since Fl depends only on n variables. The vector Zl
may not be unique, since the overlapping unknowns may have di erent choices of sub-indices. As
before, we can update the approximate values of the lth block of unknowns by exploiting the lth
block equations. Note that x ∗ = (x∗ ) is the solution of (5.1). On the other hand, we can recover
the solution x∗ from x ∗ by simply discarding the redundant overlapping parts in x ∗ . The previous
analysis now can be applied to (5.1).
x)
If F(x) is di erentiable and satis es the Lipschitz condition, then F(
will also be di erentiable
and satisfy the Lipschitz condition with the same Lipschitz constant. Condition (1.5) for F does not
When F ′ (x∗ ) is an H -matrix, however, we can show that F ′ (x ∗ )
mean a similar condition for F.
is also an H -matrix.
S
Theorem 5.1. If F ′ (x∗ ) is an M -matrix; then F ′ (x ∗ ) is also an M -matrix; where x ∗ = (x∗ );
F = (F ∗ ).
Proof. For simplicity, let A = F ′ (x∗ ); A = F ′ (x ∗ ). The diagonal entries of A are composed of the
diagonal entries of (@Fl (x ∗ ))=@xl ; l = 1; : : : ; L. Since each (@Fl (x ∗ ))=@xl is a diagonal block of A; the
diagonal entries of A are diagonal entries of A; thus positive. The nonzero o -diagonal entries of A
A = D − B; where D;
D are the
are also o -diagonal entries of A; thus nonpositive. Let A = D − B;
diagonal parts of A and A; respectively. To show A is an M -matrix, we need only to show that A−1
¡ 1; see, e.g., [23, 28]. Since A is an M -matrix,
is nonnegative. It is enough to show that (D −1 B)
−1
−1
(D B) ¡ 1; and D B is nonnegative, there exists a positive vector v ∈ R n such that D−1 Bv ¡ v.
v ¡ v. Thus (D −1 B)
¡ 1; see, e.g., [4].
Let v = (v); then we have D −1 B
Corollary 5.2. If F ′ (x∗ ) is an H -matrix; then F ′ (x ∗ ) is also an H -matrix; where x ∗ = (x∗ );
F = (F ∗ ).
Proof. Since F ′ (x∗ ) is an H -matrix, the comparison matrix of F ′ (x∗ ) is an M -matrix; see, e.g. [6,
16]. Thus by Theorem 5.1, the comparison matrix of F ′ (x ∗ ) is also an M -matrix, i.e. F ′ (x ∗ ) is an
H -matrix.
320
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
Theorem 5.1 is more general than those similar theorems in [14, 22], since the above augmentation
process allows components of x to belong to more than two subdomains.
Acknowledgements
The author would like to thank Professor Daniel B. Szyld for his careful reading of the manuscript
which improved the presentation, and also thank the referees for the valuable comments.
References
[1] Z.-Z. Bai, On the convergence of parallel chaotic nonlinear multisplitting Newton-type methods, Comput. Appl.
Math. 80 (1997) 317– 334.
[2] Z.-Z. Bai, V. Migallon, J. Penades, D.B. Szyld, Block and asynchronous two stage methods for mildly nonlinear
systems, Research Report 97-31, Dept. Math., Temple Univ., 1997, in Numer. Math., to appear.
[3] G. Baudet, Asynchronous iterative methods for multiprocessors, J. ACM 25 (1978) 226 –344.
[4] D.P. Bertsekas, J.N. Tistisklis, Parallel and Distributed Computation, Prentice-Hall, Englewood Cli s, NJ, 1989.
[5] J. Bhai, J.C. Miellou, K. Rho r, Asynchronous multisplitting methods for nonlinear xed point problems, Numer.
Algorithms 15 (1997) 315 – 345.
[6] A. Bhaya, E. Kaszkurewicz, F. Mota, Asynchronous block-iterative method for almost linear equations, Linear
Algebra Appl. 154 –156 (1991) 487– 508.
[7] A. Bojanczyk, Optimal asynchronous Newton method for the solution of nonlinear equations, J. ACM 31 (1984)
792 – 803.
[8] C.G. Broyden, A class of methods for solving nonlinear simultaneous equations, Math. Comp. 19 (1965) 577– 593.
[9] D. Chazan, W. Miranker, Chaotic relaxation, Linear Algebra Appl. 2 (1969) 199 –222.
[10] J.E. Dennis, J.J. More, Quasi-Newton methods, motivation and theory, SIAM Rev. 19 (1977) 46 – 89.
[11] J.E. Dennis, B.S. Robert, Numerical methods for unconstrained optimization and nonlinear equations, Prentice-Hall,
Englewood Cli s, NJ, 1983.
[12] D. El Baz, M -functions and parallel asynchronous algorithms, SIAM J. Numer. Anal. 27 (1990) 136 –140.
[13] M.N. El. Tarazi, Some Convergence results for asynchronous algorithms, Numer. Math. 39 (1982) 325 –340.
[14] D.J. Evans, D.-R. Wang, An asynchronous algorithm for solving a class of a nonlinear simultaneous equations,
Parallel Comput. 17 (1991) 165 –180.
[15] A. Frommer, D.B. Szyld, Asynchronous two-stage iterative methods, Numer. Math. 69 (1994) 141–153.
[16] A. Frommer, Generalized nonlinear diagonal dominance and applications to asynchronous iterative methods, Comp.
Appl. Math. 38 (1991) 105 –124.
[17] A. Frommer, H. Schwandt, D.B. Szyld, Asynchronous weighted additive Schwarz method, Electronic Trans. Numer.
Anal. 5 (1997) 48 – 61.
[18] L.-S. Kang, L.L. Sun, Y.-P. Chen, Asynchronous Parallel Methods for Problems of Mathematical Physics (Chinese),
Academic Press of China, 1985.
[19] S.-B. Kim, A. Hadjidimos, E.N. Houstis, J.R. Rice, Multi-parametrized Schwarz alterating methods for elliptic
boundary value problems, Math. Comput. Simulation 42 (1996) 47–76.
[20] H.T. Kung, Synchronized and asynchronous parallel algorithms for multiprocessors. in: J.F. Traub, (Ed.), New
Directions and Recent Results in Algorithms and Complexity, Academic Press, New York, 1976.
[21] E.S. Marwil, Convergence results for Schubert’s method for solving sparse nonlinear equations, SIAM J. Numer.
Anal. 16 (1979) 588 – 604.
[22] J.C. Miellou, D. El Baz, P. Spiteri, A new class of asynchronous iterative algorithms with order intervals, Math.
Comput. 67 (1998) 237– 225.
[23] J.M. Ortega, W.C. Rheinboldt, Iterative solution of nonlinear equations in several variables, Academic Press,
New York, 1970.
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
321
[24] L.M. Schubert, Modi cation of a quasi-Newton method for nonlinear equations with sparse Jacobian, Math. Comput.
24 (1970) 27– 30.
[25] B. Smith, P. Bj rstad, W. Gropp, Domain decomposition: Parallel multilevel methods for elliptic partial di erential
equations, Cambridge University Press, Cambridge, 1996.
[26] D.B. Szyld, Di erent models of parallel asynchronous iterations with overlapping blocks, Comput. Appl. Math. 17
(1998) 100 –115.
[27] D.B. Szyld, J.-J. Xu, Asynchronous nonlinear multisplitting methods, in prepartion.
[28] R.S. Varga, Matrix iterative analysis, Prentice-Hall, Englewood Cli s, NJ, 1962.
[29] R.S. Varga, On recurring theorems on diagonal dominance, Linear Algebra Appl. 13 (1976) 1– 9.
[30] W.P. Tang, Generalized Schwarz splittings, SIAM J. Sci. Statist. Comput. 13 (1992) 573 – 595.
[31] J.-J. Xu, An asynchronous parallel quasi-Newton methods for nonlinear equations (Chinese), in: L.-S. Kang et al.
(Eds.), Proc. Nat. Conf. on Parallel Algorithms, Zhuo-Zhou, China, 1989, pp. 113 –117.
[32] J.-J. Xu, H.-H. Wan, A class of Asynchronous block methods for nonlinear systems of equations, in: D.H. Bailey
et al. (Eds.), Proc. 7th SIAM Conf. on Parallel Processing for Scienti c Computing, SIAM, Philadelphia, 1995,
108 –109.
Convergence of partially asynchronous block
quasi-Newton methods for nonlinear systems of equations
Jian-Jun Xu ∗
Department of Mathematics, Temple University, Philadelphia, PA 19122, United States
Received 4 May 1998; received in revised form 7 October 1998
Abstract
In this paper, a partially asynchronous block Broyden method is presented for solving nonlinear systems of equations of
the form F(x) = 0. Sucient conditions that guarantee its local convergence are given. In particular, local convergence is
shown when the Jacobian F ′ (x∗ ) is an H -matrix, where x∗ is the zero point of F. The results are extended to Schubert’s
c 1999 Elsevier Science B.V. All rights
method. A connection with discrete Schwarz alternating procedure is also shown.
reserved.
Keywords: Parallel iterative methods; Asynchronous iterations; Quasi-Newton methods; Schwarz alternating procedure
1. Introduction
Chazan and Miranker rst introduced an asynchronous method for the solution of nonsingular
linear system of equations of the form Ax = b [9]. They proved that the asynchronous point method
converges to the solution if and only if A is an H -matrix [29]. This idea was extended to nonlinear
systems of equations, and there is now a considerable understanding on the convergence properties
of asynchronous iterative methods for nonlinear problems. Many authors have concentrated on xed
point problems of the form x = G(x). Convergence properties have been shown if G satis es some
contraction properties; see, e.g., [3–5, 13]. The xed point mapping G does not always arise directly
from the formulation of physical problems, however, and when it does, the contracting properties
are not always easily veri able.
In this paper, we consider the problem of nding x∗ which satis es the following nonlinear system
of equations:
F(x) = 0;
(1.1)
T
n
where F = (f1 ; : : : ; fn ) is a nonlinear operator from R into itself.
∗
E-mail: [email protected].
c 1999 Elsevier Science B.V. All rights reserved.
0377-0427/99/$ – see front matter
PII: S 0 3 7 7 - 0 4 2 7 ( 9 8 ) 0 0 2 6 8 - 4
308
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
In the literature, one can nd two general approaches for the asynchronous solutions of (1.1).
First, for an asynchronous implementation of Newton’s method, one can use two processes, one
updating the iterates, and the other evaluating the Jacobian matrix; see [3, 7, 20]. Also see [31] for
the analogue of Broyden’s method. More recently, asynchronous block methods, or more general
multisplitting methods were proposed. In [2, 6] the nonlinear term was treated explicitly because of
the weak nonlinearity. In [14, 22], convergence properties were shown where F is an M -function,
see also [12, 15] for point methods. In [12, 14, 15, 22], exact solutions for subproblems were
assumed. The use of Newton-type methods for approximate solutions of the subproblems and local
convergence theorems for the corresponding asynchronous methods were given in [1, 27, 32].
In this paper, we consider partially asynchronous block quasi-Newton methods.
Suppose F and x in (1.1) are conformally partitioned as follows:
F = (F1T ; : : : ; FLT )T ;
x = (x1T ; : : : ; xLT )T ;
where Fi : R n → Rni ; Fi = (fi1 ; : : : ; fini )T ; xi ∈ Rni ; xi = (xi1 ; : : : ; xini )T ; i = 1; : : : ; L: Let Si = {i1 ; : : : ; ini },
SL
the partition chosen is such that i=1
Si = {1; : : : ; n}; Si ∩ Sj = ∅; i 6= j; i; j = 1; : : : ; L. This partition
may correspond to a decomposition of the domain of a nonlinear partial di erential equations, see
Section 5.
The system (1.1) can be rewritten as
Fl (x1 ; : : : ; xl ; : : : ; xL ) = 0;
l = 1; : : : ; L:
(1.1)′
We consider the following nonlinear block method. Given initial values of x = (x1T ; : : : ; xLT )T , repeat
the following procedure until convergence:
For l = 1; : : : ; L
Solve for yl in Fl (x1 ; : : : ; xl−1 ; yl ; xl+1 ; : : : ; xL ) = 0;
for all l ∈ S ⊆{1; : : : ; L};
Set xl = yl for all l ∈ S:
(1.2)
In (1.2) the subset S can be chosen in a dynamic fashion. The classical nonlinear block-Jacobi
method and nonlinear block-Gauss–Seidel method [4, 23] are two examples of such selections. For
the purpose of parallel processing, the nonlinear block Jacobi method is nearly ideal, since up
to L processors can each perform one of the iterations in (1.2) with the others simultaneously.
It is synchronous in the sense that to begin the computation of the next iterate, each processor has
to wait until all processors have completed their current iteration. In practice we need to add a
synchronization mechanism to ensure that the algorithm is carried out correctly. The time necessary
to carry out the synchronization mechanism, as well as the time a processor must wait until its
data is ready to continue the calculation, adds an overhead to the computation. By removing this
synchronization and letting the processors continue their calculations according to the information
currently available, we obtain the so-called asynchronous parallel methods; see, e.g., [3, 4, 9].
Generally, the exact solutions for the subproblems are not available. Approximate methods such as
Newton’s method, Broyden’s method etc., (see, e.g., [11, 23]) can be considered to get approximate
solutions. The use of Newton-type methods were given in [1, 27, 32].
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
309
The advantages of quasi-Newton methods for (1.1) are that they involve evaluating only O(n)
scalar functions in each iteration step, which is much cheaper than Newton’s method, they do not
involve computation of the derivative of F, and they still have superlinear convergence; see, e.g. [10,
11] and the references given therein. In this paper, we consider the use of quasi-Newton methods
instead of Newton’s method. We will consider two quasi-Newton methods: Broyden’s method [8]
and Schubert’s method [24].
The main purpose of this paper is to give sucient conditions which guarantee local convergence
of partially asynchronous block quasi-Newton methods. To this end, we assume in the rest of the
paper that F and the partition satisfy the following conditions:
F is continuously di erentiable on an open convex set
⊂ R n :
(1.3)
There exists a constant ¿ 0 such that
kF ′ (x) − F ′ (x∗ )k6 kx − x∗ k;
for x ∈
:
(1.4)
All the matrices @Fi (x∗ )=@xi ; i = 1; : : : ; L; are nonsingular, moreover
(|D(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))|) ¡ 1;
(1.5)
where D(x∗ ) = diag(@F1 (x∗ )=@x1 ; : : : ; @FL (x∗ )=@xL ) is a block diagonal matrix, and we use the notation
@fi1 (x)
@xj
1
@Fi (x)
= ···
@f (x)
@xj
ini
@xj1
···
···
···
@fi1 (x)
@xjnj
···
@fini (x)
@xjnj
i; j = 1; : : : ; L;
and
@F1 (x)
@x
1
F ′ (x) =
···
@FL (x)
@x1
···
···
···
@F1 (x)
@xL
···
:
@FL (x)
@xL
Remark 1. Conditions (1.3) and (1.4) are standard. Condition (1.5) is natural for the convergence.
Consider the linear case, i.e., F(x) = Ax − b, and let F and x be partitioned as before. Suppose that
A−1 exists. Since F ′ (x) = A, condition (1.5) is necessary and sucient for the convergence of the
asynchronous block method for the linear equations F(x) = 0; see, e.g., [4, 9].
Remark 2. Condition (1.5) holds when the Jacobian matrix F ′ (x∗ ) is an H -matrix, since F ′ (x∗ ) =
D(x∗ ) − (D(x∗ ) − F ′ (x∗ )) is an H -splitting of F ′ (x∗ ); see e.g., [5, 15]. On the other hand, if each
block has only one component, i.e., if nl = 1; l = 1; : : : ; L, then (1.5) is equivalent to F ′ (x∗ ) being an
310
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
H -matrix. The class of H -matrices includes M -matrices, and also the class of strictly or irreducibly
diagonally dominant matrices [29].
In Section 2, we present both the computational and mathematical models of partially asynchronous
block Broyden’s method. The local convergence theorem is proved in Section 3 under conditions
(1.3)–(1.5). In Section 4, we point out that local convergence is still guaranteed if Schubert’s method
[24] is used instead of Broyden’s method. The connection of the block method with the Schwarz
alternating procedure (see, e.g., [18, 19, 25, 30]) is discussed in Section 5.
The weighted maximum norms is used as an important tool in proofs of asynchronous iterative
methods; see, e.g., [3, 4, 6, 9, 13, 17, 26, 27]. Given an vector w ∈ R n ; w ¿ 0. Let w = (w1T ; : : : ; wLT )T
be partitioned conformally with x, then we de ne
)
(
|xij |
kxkw = max{kxi kwi ; 16i 6L} = max
; 16j 6ni ; 16i6L :
wij
Given a matrix A ∈ R n×n . Let A = (Aij )L × L be partitioned conformally with F ′ (x), then the norms
k · kw ; k · kwi induce the following matrix norm:
kAxkw
kAkw = max
: x 6= 0; x ∈ R n :
kxkw
We extend the weighted maximum norms for rectangular matrices as follows:
k(Ai1 ; : : : ; AiL )xkwi
: x 6= 0; x ∈ R n
k(Ai1 ; : : : ; AiL )kwi = max
kxkw
and
(
kAij xj kwi
kAij kwi = max
: xj 6= 0; xj ∈ R nj
kxj kwj
)
:
We immediately have
kAii (Ai1 ; : : : ; AiL )kwi 6kAii kwi · k(Ai1 ; : : : ; AiL )kwi ;
(1.6)
k|A|kw = kAkw ;
(1.7)
k|(Ai1 ; : : : ; AiL )|kwi = k(Ai1 ; : : : ; AiL )kwi ;
kAkw = max{k(Ai1 ; : : : ; AiL )kwi ; 16i 6L};
(1.8)
kAij kwi 6k(Ai1 ; : : : ; AiL )kwi ;
(1.9)
and
16i; j 6L:
By the theory of nonnegative matrix (see, e.g., [4, 28]), condition (1.5) is equivalent to
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
311
(1:5)′ All matrices @Fi (x∗ )=@xi ; i = 1; : : : ; L; are nonsingular, moreover, there exists 0 ¡ 1, and a
vector w ¿ 0 such that
kD(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))kw ¡ 0 :
2. Partially asynchronous block Broyden method
Broyden’s method [8] for (1.1) is as follows:
x(k + 1) = x(k) − B(k)−1 F(x(k));
B(k + 1) = B(k) + (1=z(k)T z(k)) · (y(k) − B(k)z(k))z(k)T ;
where y(k) = F(x(k + 1)) − F(x(k));
k = 1; 2; : : : ;
z(k) = x(k + 1) − x(k);
where x(0) and B(0) are initial approximation to the solution x∗ and initial approximation the
Jacobian F ′ (x∗ ), respectively.
We will rst describe a computational model for partially asynchronous block Broyden’s method,
i.e., how the computer is actually programmed to execute its instructions, then we give a mathematics
model, i.e., how the execution is described mathematically in order to analyze convergence [26].
Let us as before assume that there are L processors. We design L processes. Each processor
perform computation in one process. In each process, say the lth process, we use one step of
Broyden’s method to update the lth block xl in the lth subproblem in (1.2), l = 1; : : : ; L. More
precisely, the computational model can be written as the following pseudo-code:
Given an initial guess x, and initial matrices Bl ; l = 1; : : : ; L.
Process l(l = 1; : : : ; L)
Initialization for x; Bl
Step 1: Compute Fl := Fl (x);
Step 2: Set x := xl ; Fl := Fl ;
Step 3: Compute xl := xl − Bl−1 Fl ;
Step 4: Send xl to the other processes;
Step 5: Check the termination criterion, if satis ed then stop;
Step 6: Receive the values of x (except xl ) from the other processes;
Step 7: Compute Fl := Fl (x);
Step 8: Compute z := xl − x l ; y := Fl − Fl ;
Step 9: Compute Bl := Bl + (1=z T z) · (y − Bl z)z T ;
Step 10: Goto step 2;
where Bl ; F l ; x;
y; z are local variables, only x is a global variable. In each cycle of a process, say
the lth process, one step of Broyden’s update is performed for a subproblem of nl dimension. The
L processes can be done in parallel. There is no explicit synchronization among the processes.
Remark 3. Since Broyden update is a rank-one update, the Sherman–Morrison formula can be used
to update the inverse of Bl . Let Hl = Bl−1 , then initialization for Bl is changed for Hl , Steps 3 and 9
are rewritten as follows while the other steps remain unchanged:
Step 3′ : Compute xl := xl − Hl · Fl ;
Step 9′ : Hl := Hl + (1=z T Hl y) · (y − Bl z)z T ;
312
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
One can also uses QR factorization technique to solve the linear system in step 3; see, e.g., [11].
We give the following mathematical model in order to analyze the convergence.
Let the initial approximations x(0) = (x1 (0)T ; : : : ; xL (0)T )T , and Bi (0); i = 1; : : : ; L be given, then for
k = 0; 1; : : : ; i = 1; : : : ; L;
−1
xi (k + 1) = xi (si (k)) − Bi (k) Fi (u(k)) if i ∈ I (k)
else
xi (k + 1) = xi (k)
Bi (k + 1) = Bi (k) + (1=zi (k)T zi (k)) · (yi (k) − Bi (k)zi (k))zi (k)T
Bi (k + 1) = Bi (k)
if i ∈ I (k)
else;
where u(k) = (x1 (s1 (k))T ; : : : ; xL (sL (k))T )T ;
yi (k) = Fi (x(k + 1)) − Fi (v(k; i));
v(k; i) = (x1 (k + 1)T ; : : : ; xi−1 (k + 1)T ;
xi (si (k))T ; xi+1 (k + 1)T ; : : : ; xL (k + 1)T )T ;
zi (k) = xi (k + 1) − xi (si (k));
where I (k) and sj (k) satisfy the following conditions
(2.1) for some xed integer p ¿ 0; k − p ¡ sj (k)6k; j = 1; : : : ; L; k = 0; 1; : : :
(2.2) each I (k); k = 0; 1; : : : are subset of {1; : : : ; L}, and the set {k: j ∈ I (k)} is unbounded for
all j = 1; : : : ; L.
Conditions (2.1) and (2.2) may be interpreted as follows: At each instant of time k, all the ith
block component xi ; i ∈ I (k), are updated while the remaining are unchanged. The updating uses the
rst block component of x(s1 (k)), the second block component of x(s2 (k)), etc. Then the matrices
Bi ; i ∈ I (k), are updated by using the new results. Each pair of block component and matrix is
updated in nitely many often, and no update uses a value of a component which was produced by
an update p or more steps previously.
The assumption of the existence of uniform bound p for delays appeared in the rst asynchronous
model in [9]. It is not an additional constraint in most practical implementations [3]. This type of
asynchronous model was called partially asynchronous method in [4].
We will refer the above mathematical model as partially asynchronous block Broyden’s (PABB)
method.
If we take si (k) = k; I (k) = {1; : : : ; L} for all k and i, then PABB method describes block-Jacobi
Broyden method.
If we take si (k) = k; I (k) = {k + 1 (mod L)} for all k, then PABB method describes block-Gauss–
Seidel Broyden method.
3. Local convergence theorem
In this section we prove the local convergence of the PABB method. We have:
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
313
Theorem 3.1. Suppose that (1:3)–(1:5) are satis ed; then there exist two constants ¿ 0; ¿ 0;
such that if kx(0) − x∗ kw 6 for w ¿ 0 de ned in (1:5)′ ; kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L;
then the sequence {x(k)} generated by PABB method converges to x∗ .
Before we give the proof of Theorem 3.1, we need several Lemmas. Because of the norm equivalence, we may assume that the norm in (1.4) is k · kw , and there exist two constants 1 ; 2 ¿ 0
such that
kAk2 61 kAkwˆ ;
kAkwˆ 62 kAk2 ;
(3.1)
where wˆ = w if A ∈ Rn × n , and wˆ = wi if A ∈ Rni × n (or Rni × nj ).
Lemma 3.2 (Ortega and Rheinboldt [23]). If F : R n → Rm is Gateaux-di erentiable in an open convex set
0 ; then
F(y) − F(x) =
Z
1
F ′ (x + t(y − x))(y − x) dt
for x; y ∈
0 :
0
Lemma 3.3 (Ortega and Rheinboldt [23]). Let A; C ∈ Rn × n ; A is nonsingular and kA−1 k2 6 1 ; kA−
Ck2 6 2 ; 1 2 ¡ 1; then C is also nonsingular; moreover;
kC −1 k2 6 1 =(1 − 1 2 ):
Lemma 3.4. Under the conditions (1:3)–(1:5); we have
@F (x)
@Fi (x∗ )
i
−
6 kx − x∗ kw
@xi
@xi
for x ∈
(3.2)
wi
and there exists S(x∗ ; 1 ) = {x: kx − x∗ k61 } ⊂
; such that
!−1
@Fi (x∗ )
@Fi (x)
@Fi (x) @Fi (x)
@Fi (x)
¡ 0 ;
;:::;
; 0;
;:::;
@xi
@x1
@xi−1
@xi+1
@xL
i = 1; : : : ; L; x ∈ S(x∗ ; 1 );
wi
xi − x∗ −
i
@Fi (x∗ )
Fi (x)
@xi
wi
!
∗ ) −1
@F
(x
1
i
· kx − x∗ k2 ;
6kx − x∗ kw + ·
w
2
@xi
(3.3)
!−1
i = 1; : : : ; L; x ∈ S(x∗ ; 1 ):
(3.4)
wi
Proof. By the properties of the norms k · kw ; k · kwi in Section 1 and (1.4), we immediately have
(3.2). Since kD(x∗ )−1 (F ′ (x∗ ) − D(x∗ ))kw ¡ 0 , and F ′ (x) is continuously di erentiable at x∗ , there
314
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
exists S(x∗ ; 1 ) = {x: kx − x∗ kw 61 } ⊂
; such that
kD(x∗ )−1 (F ′ (x) − D(x))kw ¡ 0 ; for x ∈ S(x∗ ; 1 )
thus by (1.8), (3.3) holds.
By Lemma 3.2, (3.2) and (3.3), we have
@Fi (x∗ )
@xi
kxi − xi∗ −
!−1
Fi (x)kwi
!−1
!
∗)
@Fi (x∗ )
@F
(x
i
∗
∗
Fi (x) − Fi (x ) −
(xi − xi )
=
@xi
@xi
wi
!−1 "Z
1
@Fi (x∗ )
@Fi (x∗ + t(x − x∗ ))
@Fi (x∗ + t(x − x∗ ))
=
;:::;
; 0;
@xi
@x1
@xi−1
0
!
@Fi (x∗ + t(x − x∗ ))
@Fi (x∗ + t(x − x∗ ))
(x − x∗ ) dt
;:::;
@xi+1
@xL
+
Z
1
@F (x∗ + t(x − x∗ ))
i
0
@xi
−
@F (x∗ )
i
@xi
!
#
∗
(xi − xi ) dt
wi
!
∗ ) −1
1
@F
(x
i
· kx − x∗ k2
60 kx − x∗ kw +
w
2
@xi
for x ∈ S(x∗ ; 1 ):
wi
Lemma 3.5 (Dennis and Robert [11]). Let u; v ∈ R n ; uT v = 1; then
kI − uvT k2 = kuk2 · kvk2 :
Lemma 3.6. Let i ∈ I (k); then
@Fi (x∗ )
@Fi (x∗ )
Bi (k + 1) −
6
Bi (k) −
+ 1 (x(k + 1); v(k; i));
@xi
2
@xi
2
where (x(k + 1); v(k; i)) = max{kx(k + 1) − x∗ k2 ; kv(k; i) − x∗ k2 }; 1 = 1 2 .
Proof. If i ∈ I (k), from Lemma 3.5 we have
@Fi (x∗ )
Bi (k + 1) −
@xi
2
!
∗
(yi − @Fi (x ) zi (k))zi (k)T
zi (k)zi (k)T
@Fi (x∗ )
@xi
I−
6
Bi (k) −
+
@xi
zi (k)T zi (k)
2
zi (k)T zi (k)
2
∗
∗
@Fi (x )
1
@Fi (x )
zi (k)
;
6
Bi (k) −
+
yi (k) −
@xi
kzi (k)k2
@xi
2
2
315
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
while by (3.1), (3.2)
@Fi (x∗ )
zi (k)
yi (k) −
@xi
2
@Fi (x∗ )
(xi (k + 1) − xi (si (k)))
=
Fi (x(k + 1)) − Fi (v(k; i)) −
@xi
2
Z
1 @F (v(k; i) + t(x(k + 1) − v(k; i)))
∗
@Fi (x )
i
−
)(xi (k + 1) − xi (si (k))) dt
=
(
0
@xi
@xi
2
6 1 max{kx(k + 1) − x∗ k2 ; kv(k; i) − x∗ k2 }kxi (k + 1) − xi (si (k))k2 ;
thus we have proved Lemma 3.6.
Proof of Theorem 3.1. Suppose that for some xed ¿ 0; we have
!−1
@Fi (x∗ )
6 ;
@xi
kF ′ (x∗ )kw 6 ;
i = 1; : : : ; L:
(3.5)
wi
For any 2 ¿ 0, by Lemma 3.3, there exists ¿ 0 such that if kBi − (@Fi (x∗ ))=@xi k2 62, then Bi
is nonsingular and
−1
B −
i
@F (x∗ )
i
@xi
!−1
62 ;
i = 1; : : : ; L:
(3.6)
2
Let kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L; kx(0) − x∗ kw 6; 61 . We can choose 2 and small
enough, such that
1
1
r ≡ 2 2 ( + 2 ) + 2 + 0 ¡ 1;
2p
1 1 6;
(3.7)
1−r
We use mathematical induction to show the following inequalities:
kxi (k + 1) − xi∗ kwi 6rkx(s(k)) − x∗ kw if i ∈ I (k);
kx (k + 1) − x∗ k 6 kx(k) − x∗ k else;
i
i
wi
w
@Fi (x∗ )
Bi (k + 1) −
62
@xi
(3.8)
2
and as a byproduct we get
kx(k) − x∗ kw 6r q−1 kx(0) − x∗ kw ;
if k ¿ (q − 1) · 2p:
(3.9)
316
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
If i ∈ I (0), by (3.1), (3.4)–(3.6) we have
kxi (1) − xi∗ kwi = kxi (si (0)) − xi∗ − Bi (0)−1 Fi (u(0))kwi
@F (x∗ ) −1
i
−1
− Bi (0)
· kFi (u(0))kwi
6
@xi
w
i
∗
+
xi (si (0)) − xi −
@Fi (x∗ )
@xi
!−1
Fi (u(0))
wi
1
6 2 2 kFi (u(0))kwi + 0 ku(0) − x∗ kw + ku(0) − x∗ k2w :
2
By Lemma 3.2,
kFi (u(0))kwi = kFi (u(0)) − Fi (x∗ )kwi
6 kF(u(0)) − F(x∗ ) − F ′ (x∗ )(u(0) − x∗ )kw + kF ′ (x∗ )(u(0) − x∗ )kw
1
6 ku(0) − x∗ k2w + ku(0) − x∗ kw :
2
Since u(0) = x(0), thus
1
1
kxi (1) − xi∗ kwi 6 2 2
+ + 0 + kx(0) − x∗ kw
2
2
∗
= rkx(0) − x k :
w
If i ∈= I (0), then
kxi (1) − xi∗ kwi = kxi (0) − xi∗ kwi 6kx(0) − x∗ kw :
On the other hand, if i ∈ I (0), by Lemma 3.6
@Fi (x∗ )
@Fi (x∗ )
Bi (1) −
+ 1 (x(1); v(0; i))
6
Bi (0) −
@xi
@xi
2
2
6 + 1 1 kx(0) − x∗ kw 6 + 1 1 62:
If i ∈= I (0),
@Fi (x∗ )
@Fi (x∗ )
Bi (1) −
=
Bi (0) −
62:
@xi
@xi
2
2
Assume that (3.8) holds for k = 0; 1; : : : ; m − 1. It is clear that kx(k) − x∗ kw 6; k = 0; 1; : : : ; m; Bi (k)
is nonsingular, moreover, by (3.6), k(Bi (k))−1 − (@Fi (x∗ ))=@xi )−1 k2 62 ; i = 1; : : : ; L; k = 0; 1; : : : ; m
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
317
Thus if i ∈ I (m),
kxi (m + 1) − xi∗ kwi = kxi (si (m)) − xi∗ − Bi (m)−1 Fi (u(m))kwi
!
∗ ) −1
@F
(x
i
−1
· kFi (u(m))kw
6
i
Bi (m) −
@xi
wi
!
∗ ) −1
@F
(x
i
∗
+
xi (si (m)) − xi −
Fi (u(m))
@xi
wi
1
1
+ ku(m) − x∗ kw + 0 + ku(m) − x∗ kw
2
2
∗
∗
= rku(m) − x kw 6rkx(s(m)) − x kw ;
6 2 2
where s(m) is an element in {si (m); i = 1; : : : ; L}, thus m − p ¡ s(m)6m.
If i ∈= I (m), we have kxi (m + 1) − xi∗ kwi = kxi (m) − xi∗ kwi 6kx(m) − x∗ kw .
Assume that for some h; (h − 1) · 2p ¡ m6h · 2p.
We will show that (3.9) holds for if (q − 1) · 2p ¡ k 6m + 1; 16q6h.
We have already shown (3.9) for q = 1. Assume that it holds for 16q6 q ¡ h. Because of
the uniform bound p for the delays, each block component is updated at least once for every 2p
consecutive iterations, say (k + 1)th; : : : ; (k + 2p)th iterations, and the values of the block components
used for the updating all come from these 2p iterates. Thus for any k; q · 2p ¡ k 6m + 1, and
any i ∈ {1; : : : ; L}, there exists k1 6k − 1; u(k1 ) = (x1 (s1 (k1 ))T ; : : : ; xL (sL (k1 ))T )T ; si (k1 ) ¿ (q − 1) ·
2p; i = 1; : : : ; L; such that
kxi (k) − xi∗ kwi = kxi (k1 + 1) − xi∗ kwi
= kxi (si (k1 )) − xi∗ − Bi (k1 )−1 Fi (u(k1 ))kwi 6rku(k1 ) − x∗ kw
6r · r (q−2)
kx(0) − x∗ k = r (q−1)
kx(0) − x∗ k :
w
w
Thus kx(k) − x∗ kw 6r (q−1)
kx(0) − x∗ kw :
Therefore (3.9) holds for k 6m + 1. Thus if i ∈ I (m); by Lemma 3.6,
@Fi (x∗ )
@Fi (x∗ )
6
Bi (m) −
+ 1 (x(m + 1); v(m; i))
Bi (m + 1) −
@xi
2
@xi
2
@Fi (x∗ )
6
Bi (m) −
+ 1 1 r h−3
@xi
2
@Fi (x∗ )
6 · · · 6
Bi (m − 2p + 1) −
+ 2p1 1 r h−3
@xi
2
@Fi (x∗ )
6 · · · 6
Bi (0) −
+ 2p1 1 (r h−3 + · · · + r + 1)
@xi
2
2p1
6+
62:
1−r
318
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
If i ∈= I (m);
@Fi (x∗ )
@Fi (x∗ )
=
Bi (m) −
62:
Bi (m + 1) −
@xi
@xi
2
2
Therefore (3.8) holds for all k; and then (3.9) holds for all k; thus {x(k)} convergences to x∗ .
By Remark 2, we have the following corollary.
Corollary 3.7. Suppose that (1.3)–(1.4) are satis ed; and F ′ (x∗ ) is an H -matrix; then for any
partitioning of F and x; there exist two constants ¿ 0; ¿ 0; such that if kx(0) − x∗ kw 6 for
w ¿ 0 de ned in (1.5)′ ; kBi (0) − (@Fi (x∗ ))=@xi k2 6; i = 1; : : : ; L; the sequence {x(k)} generated by
PABB method converges to x∗ .
4. Extension to Schubert’s method
Schubert’s method [24] is a variation of Broyden’s method which is of interest in the case that
the Jacobian is sparse. In this variation, the approximate Jacobian is forced to have the same sparsity
pattern as the Jacobian.
Given a vector function F = (f1 ; : : : ; fm )T ∈ Rm → Rm ; we de ne its sparsity pattern as a matrix
PF = (pij )m×m ; where
pij =
0 if fi (x) is independent of xj
1 else
i; j = 1; : : : ; m:
Let PFl = (plij ) be the sparsity pattern of Fl ; l = 1; : : : ; L. Using Schubert’s method instead of Broyden’s method for each subproblem in (1.2) leads to the following computational model:
Given initial solution x = (x1T ; : : : ; xLT )T ; and matrices Bl .
Process l(l = 1; : : : L)
Initialization for x = (x1T ; : : : ; xlT ; : : : ; xLT )T ; Bl
Steps 1–8 and 10 are the same as the computational model of PABB method;
step 9: if plij = 1 then uij ←zj else uij ←0; i; j = 1; : : : ; nl ;
Bl ← Bl +
nl
X
i=1
ui 6=0
ei eiT
(y − Bl z) T
ui ;
uiT ui
where ui = (ui1 ; : : : ; uinl )T ; i = 1; : : : ; nl ; and ei ∈ Rnl is the unit vector of which the ith component is
1, i = 1; : : : ; nl .
Its mathematical model can be described similarly as PABB method.
Also its local convergence can be shown in an analogous way to PABB method, since we can
easily get an inequality for Schubert’s method similar to Lemma 3.6, see [21] ((3.14), p. 593).
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
319
5. Connection with the discrete Schwarz alternating procedure
In this section, suppose that there are overlaps among the blocks x1 ; : : : ; xL ; and correspondingly
among F1 ; : : : ; FL . This may happen when discrete Schwarz alternating procedure (see, e.g., [18, 19,
25, 30]) is used to solve (1.1), where each block xl corresponds to the unknowns in a subdomain.
In this case, the computational models need not change, but we need to modify the mathematical
model so that thePprevious analysis can be applied.
Suppose that Ll= 1 nl = m ¿ n; we de ne a vector x = (x T1 ; : : : ; x TL )T ∈ Rm from x ∈ R n by setting
x l = xl ; l = 1; : : : ; L: Denote this mapping from R n to Rm by x = (x), similarly we obtain F =
(F T1 ; : : : ; F TL )T = (F). We can thus rewrite (1.1) by repeating the overlapping unknowns and the
corresponding equations:
Fl (x l ; Zl ) = 0;
l = 1; : : : ; L;
(5.1)
where the vector Zl is a subset of i6=l {x l }; since Fl depends only on n variables. The vector Zl
may not be unique, since the overlapping unknowns may have di erent choices of sub-indices. As
before, we can update the approximate values of the lth block of unknowns by exploiting the lth
block equations. Note that x ∗ = (x∗ ) is the solution of (5.1). On the other hand, we can recover
the solution x∗ from x ∗ by simply discarding the redundant overlapping parts in x ∗ . The previous
analysis now can be applied to (5.1).
x)
If F(x) is di erentiable and satis es the Lipschitz condition, then F(
will also be di erentiable
and satisfy the Lipschitz condition with the same Lipschitz constant. Condition (1.5) for F does not
When F ′ (x∗ ) is an H -matrix, however, we can show that F ′ (x ∗ )
mean a similar condition for F.
is also an H -matrix.
S
Theorem 5.1. If F ′ (x∗ ) is an M -matrix; then F ′ (x ∗ ) is also an M -matrix; where x ∗ = (x∗ );
F = (F ∗ ).
Proof. For simplicity, let A = F ′ (x∗ ); A = F ′ (x ∗ ). The diagonal entries of A are composed of the
diagonal entries of (@Fl (x ∗ ))=@xl ; l = 1; : : : ; L. Since each (@Fl (x ∗ ))=@xl is a diagonal block of A; the
diagonal entries of A are diagonal entries of A; thus positive. The nonzero o -diagonal entries of A
A = D − B; where D;
D are the
are also o -diagonal entries of A; thus nonpositive. Let A = D − B;
diagonal parts of A and A; respectively. To show A is an M -matrix, we need only to show that A−1
¡ 1; see, e.g., [23, 28]. Since A is an M -matrix,
is nonnegative. It is enough to show that (D −1 B)
−1
−1
(D B) ¡ 1; and D B is nonnegative, there exists a positive vector v ∈ R n such that D−1 Bv ¡ v.
v ¡ v. Thus (D −1 B)
¡ 1; see, e.g., [4].
Let v = (v); then we have D −1 B
Corollary 5.2. If F ′ (x∗ ) is an H -matrix; then F ′ (x ∗ ) is also an H -matrix; where x ∗ = (x∗ );
F = (F ∗ ).
Proof. Since F ′ (x∗ ) is an H -matrix, the comparison matrix of F ′ (x∗ ) is an M -matrix; see, e.g. [6,
16]. Thus by Theorem 5.1, the comparison matrix of F ′ (x ∗ ) is also an M -matrix, i.e. F ′ (x ∗ ) is an
H -matrix.
320
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
Theorem 5.1 is more general than those similar theorems in [14, 22], since the above augmentation
process allows components of x to belong to more than two subdomains.
Acknowledgements
The author would like to thank Professor Daniel B. Szyld for his careful reading of the manuscript
which improved the presentation, and also thank the referees for the valuable comments.
References
[1] Z.-Z. Bai, On the convergence of parallel chaotic nonlinear multisplitting Newton-type methods, Comput. Appl.
Math. 80 (1997) 317– 334.
[2] Z.-Z. Bai, V. Migallon, J. Penades, D.B. Szyld, Block and asynchronous two stage methods for mildly nonlinear
systems, Research Report 97-31, Dept. Math., Temple Univ., 1997, in Numer. Math., to appear.
[3] G. Baudet, Asynchronous iterative methods for multiprocessors, J. ACM 25 (1978) 226 –344.
[4] D.P. Bertsekas, J.N. Tistisklis, Parallel and Distributed Computation, Prentice-Hall, Englewood Cli s, NJ, 1989.
[5] J. Bhai, J.C. Miellou, K. Rho r, Asynchronous multisplitting methods for nonlinear xed point problems, Numer.
Algorithms 15 (1997) 315 – 345.
[6] A. Bhaya, E. Kaszkurewicz, F. Mota, Asynchronous block-iterative method for almost linear equations, Linear
Algebra Appl. 154 –156 (1991) 487– 508.
[7] A. Bojanczyk, Optimal asynchronous Newton method for the solution of nonlinear equations, J. ACM 31 (1984)
792 – 803.
[8] C.G. Broyden, A class of methods for solving nonlinear simultaneous equations, Math. Comp. 19 (1965) 577– 593.
[9] D. Chazan, W. Miranker, Chaotic relaxation, Linear Algebra Appl. 2 (1969) 199 –222.
[10] J.E. Dennis, J.J. More, Quasi-Newton methods, motivation and theory, SIAM Rev. 19 (1977) 46 – 89.
[11] J.E. Dennis, B.S. Robert, Numerical methods for unconstrained optimization and nonlinear equations, Prentice-Hall,
Englewood Cli s, NJ, 1983.
[12] D. El Baz, M -functions and parallel asynchronous algorithms, SIAM J. Numer. Anal. 27 (1990) 136 –140.
[13] M.N. El. Tarazi, Some Convergence results for asynchronous algorithms, Numer. Math. 39 (1982) 325 –340.
[14] D.J. Evans, D.-R. Wang, An asynchronous algorithm for solving a class of a nonlinear simultaneous equations,
Parallel Comput. 17 (1991) 165 –180.
[15] A. Frommer, D.B. Szyld, Asynchronous two-stage iterative methods, Numer. Math. 69 (1994) 141–153.
[16] A. Frommer, Generalized nonlinear diagonal dominance and applications to asynchronous iterative methods, Comp.
Appl. Math. 38 (1991) 105 –124.
[17] A. Frommer, H. Schwandt, D.B. Szyld, Asynchronous weighted additive Schwarz method, Electronic Trans. Numer.
Anal. 5 (1997) 48 – 61.
[18] L.-S. Kang, L.L. Sun, Y.-P. Chen, Asynchronous Parallel Methods for Problems of Mathematical Physics (Chinese),
Academic Press of China, 1985.
[19] S.-B. Kim, A. Hadjidimos, E.N. Houstis, J.R. Rice, Multi-parametrized Schwarz alterating methods for elliptic
boundary value problems, Math. Comput. Simulation 42 (1996) 47–76.
[20] H.T. Kung, Synchronized and asynchronous parallel algorithms for multiprocessors. in: J.F. Traub, (Ed.), New
Directions and Recent Results in Algorithms and Complexity, Academic Press, New York, 1976.
[21] E.S. Marwil, Convergence results for Schubert’s method for solving sparse nonlinear equations, SIAM J. Numer.
Anal. 16 (1979) 588 – 604.
[22] J.C. Miellou, D. El Baz, P. Spiteri, A new class of asynchronous iterative algorithms with order intervals, Math.
Comput. 67 (1998) 237– 225.
[23] J.M. Ortega, W.C. Rheinboldt, Iterative solution of nonlinear equations in several variables, Academic Press,
New York, 1970.
J.-J. Xu / Journal of Computational and Applied Mathematics 103 (1999) 307–321
321
[24] L.M. Schubert, Modi cation of a quasi-Newton method for nonlinear equations with sparse Jacobian, Math. Comput.
24 (1970) 27– 30.
[25] B. Smith, P. Bj rstad, W. Gropp, Domain decomposition: Parallel multilevel methods for elliptic partial di erential
equations, Cambridge University Press, Cambridge, 1996.
[26] D.B. Szyld, Di erent models of parallel asynchronous iterations with overlapping blocks, Comput. Appl. Math. 17
(1998) 100 –115.
[27] D.B. Szyld, J.-J. Xu, Asynchronous nonlinear multisplitting methods, in prepartion.
[28] R.S. Varga, Matrix iterative analysis, Prentice-Hall, Englewood Cli s, NJ, 1962.
[29] R.S. Varga, On recurring theorems on diagonal dominance, Linear Algebra Appl. 13 (1976) 1– 9.
[30] W.P. Tang, Generalized Schwarz splittings, SIAM J. Sci. Statist. Comput. 13 (1992) 573 – 595.
[31] J.-J. Xu, An asynchronous parallel quasi-Newton methods for nonlinear equations (Chinese), in: L.-S. Kang et al.
(Eds.), Proc. Nat. Conf. on Parallel Algorithms, Zhuo-Zhou, China, 1989, pp. 113 –117.
[32] J.-J. Xu, H.-H. Wan, A class of Asynchronous block methods for nonlinear systems of equations, in: D.H. Bailey
et al. (Eds.), Proc. 7th SIAM Conf. on Parallel Processing for Scienti c Computing, SIAM, Philadelphia, 1995,
108 –109.