The Cauchy–FFT Method

3.2.2 The Cauchy–FFT Method

An alternative method for deriving coefficients of power series when many terms are needed is based on the following classic result. Suppose that the value f (z) of an analytic function can be computed at any point inside and on the circle C r = {z : |z − a| = r}, and set

M(r) = max |f (z)|, z = a + re iθ ∈C r . Then the coefficients of the Taylor expansion around a are determined by Cauchy’s

iθ )e −niθ dθ. (3.2.8) 2πi C r (z

For a derivation, multiply the Taylor expansion (3.1.3) by (z − a) −n−1 , integrate term by term over C r , and note that

(z − a) j −n−1 dz =

r j −n e (j −n)iθ dθ =

1 if j = n, (3.2.9)

2πi C r

From the definitions and (3.2.8) it follows that

|a n |≤r −n M(r).

194 Chapter 3. Series, Operators, and Continued Fractions Further, with z ′ =a+r ′ e iθ ,0≤r ′ <r , we have

M(r)(r ′ /r) n |R n (z ) |≤

|a j (z − a) |≤

r −j M(r)(r ′ ) =

/r This form of the remainder term of a Taylor series is useful in theoretical studies, and also

for practical purpose, if the maximum modulus M(r) is easier to estimate than the nth derivative.

Set 4θ = 2π/N, and apply the trapezoidal rule (see (1.1.12)) to the second integral in (3.2.8). Note that the integrand has the same value for θ = 2π as for θ = 0. The terms

2 f 0 and 1 2 f N that appear in the general trapezoidal rule can therefore in this case be replaced by f 0 . Then

1 N −1

a n ik4θ ≈ ˜a n ≡ −ink4θ n f (a + re )e , n Nr

= 0 : N − 1. (3.2.12)

k =0

The approximate Taylor coefficients ˜a n , or rather the numbers a ⋆ n = ˜a n Nr n , are here ex- pressed as a case of the (direct) discrete Fourier transform (DFT). More generally, this transform maps an arbitrary sequence {α N

0 to a sequence {a n } −1 0 , by the following equations:

k } −1

N −1

α k e −ink4θ , n = 0 : N − 1.

k =0

It will be studied more systematically in Sec. 4.6.2. If N is a power of 2, it is shown in Sec. 4.7 that, given the N values α k , k = 0 : N −1, and e −i4θ , no more than N log 2 N complex multiplications and additions are needed for the computation of all the N coefficients a ⋆ n , if an implementation of the DFT known as the fast Fourier transform (FFT) is used. This makes our theoretical considerations very practical.

It is also shown in Sec. 4.7 that the inverse of the DFT (3.2.13) is given by the formulas

N −1

α k = (1/N)

a n ⋆ e ink4θ , k = 0 : N − 1. (3.2.14)

n =0

This looks almost like the direct DFT (3.2.13), except for the sign of i and the factor 1/N. It can therefore also be performed by means of O(N log N) elementary operations, instead

of the O(N 3 ) operations that the most obvious approach to this task would require (i.e., by solving the linear system (3.2.13)). In our context, i.e., the computation of Taylor coefficients, we have, by (3.2.12) and the line after that equation,

(3.2.15) Set z k

= a + re ik4θ . Using (3.2.15), the inverse transformation then becomes 58

N −1

f (z n

˜a n (z k − a) , k = 0 : N − 1. (3.2.16)

n =0

n −1 =0 ˜a n (z − a) is the solution of a special, although important, interpolation problem for the function f , analytic inside a circle in C.

58 One interpretation of these equations is that the polynomial

3.2. More about Series 195 Since the Taylor coefficients are equal to f (n) (a)/n !, this is de facto a method for

the accurate numerical differentiation of an analytic function. 59 If r and N are chosen appropriately, it is more well-conditioned than most methods for numerical differentiation, such as the difference approximations mentioned in Chapter 1; see also Sec. 3.3. It requires, however, complex arithmetic for a convenient implementation. We call this the Cauchy– FFT method for Taylor coefficients and differentiation.

The question arises of how to choose N and r. Theoretically, any r less than the radius of convergence ρ would do, but there may be trouble with cancellation if r is small. On the other hand, the truncation error of the numerical integration usually increases with

r . Scylla and Charybdis situations 60 like this are very common with numerical methods. Typically it is the rounding error that sets the limit for the accuracy ; it is usually not expensive to choose r and N such that the truncation error becomes much smaller. A rule of thumb for this situation is to guess a value of ˆ N , i.e., how many terms will be needed in the expansion, and then to try two values for N (powers of 2) larger than ˆ N . If ρ is finite try r = 0.9ρ and r = 0.8ρ, and compare the results. They may or may not indicate that some

other values of N and r should also be tried. On the other hand, if ρ = ∞ try, for example, r = 1 and r = 3, and compare the results. Again, the results indicate whether or not more experiments should be done.

One can also combine numerical experimentation with a theoretical analysis of a more or less simplified model, including a few elementary optimization calculations. The authors take the opportunity to exemplify below this type of “hard analysis” on this question.

We first derive two lemmas, which are important also in many other contexts. First we have a discrete analogue of (3.2.9).

Lemma 3.2.2.

Let p and N be integers. Then

N −1

e 2πipk/N = 0,

k =0

unless p = 0 or p is a multiple of N. In these exceptional cases every term equals 1, and the sum equals N.

Proof. If p is neither zero nor a multiple of N, the sum is a geometric series, the sum of which is equal to

(e 2πip

− 1)/(e 2πip/N − 1) = 0.

The rest of the statement is obvious. We next show an error estimate for the approximation provided by the trapezoidal

rule (3.2.12).

59 The idea of using Cauchy’s formula and FFT for numerical differentiation seems to have been first suggested by Lyness and Moler [252]; see Henrici [194, Sec. 3].

60 According to the American Heritage Dictionary, Scylla is a rock on the Italian side of the Strait of Messina, opposite to the whirlpool Charybdis, personified by Homer (Ulysses) as a female sea monster who devoured sailors.

The problem is to navigate safely between them.

196Chapter 3. Series, Operators, and Continued Fractions

Lemma 3.2.3.

− a) n is analytic in the disk |z − a| < ρ. Let ˜a n be defined by ( 3.2.12), where 0 < r < ρ. Then

Suppose that f (z) = ∞

0 a n (z

˜a N n −a n =a n +N r +a n +2N r 2N +a n r +3N 3N + · · · , 0 ≤ n < N. (3.2.17)

Proof. Since 4θ = 2π/N,

∞ m ˜a n

1 N −1

= 2πik/N e −2πink/N a m re

e n 2πi(−n+m)k/N .

By the previous lemma, the inner sum of the last expression is zero, unless m − n is a multiple of N. Hence (recall that 0 ≤ n < N),

˜a n =

r +2N N +··· , Nr

from which (3.2.17) follows. Lemma 3.2.3 can, with some modifications, be generalized to Laurent series (and to

complex Fourier series); for example, (3.2.17) becomes ˜c n −c n =···c n −2N r −2N

+c n +N r +c n +2N r ···. (3.2.18) Let M(r) be the maximum modulus for the function f (z) on the circle C r , and denote

+c 2N n r −N

−N

by M(r)U an upper bound for the error of a computed function value f (z), |z| = r, where U ≪ 1. Assume that rounding errors during the computation of ˜a n are of minor importance.

Then, by (3.2.12), M(r)U/r n is a bound for the rounding error of ˜a n . (The rounding errors during the computation can be included by a redefinition of U.)

Next we shall consider the truncation error of (3.2.12). First we estimate the coef- ficients that occur in (3.2.17) by means of max |f (z)| on a circle with radius r ′ ;r ′ >r ,

where r is the radius of the circle used in the computation of the first N coefficients. Thus, in (3.2.8) we substitute r ′ , j for r, n, respectively, and obtain the inequality

|a j | ≤ M(r ′ )(r ′ ) −j , 0<r<r ′ < ρ.

The actual choice of r ′ strongly depends on the function f . (In rare cases we may choose r ′ = ρ.) Put this inequality into (3.2.17), where we shall choose r < r ′ <ρ . Then

| ˜a n −a n | ≤ M(r ′ ) (r ′ ) −n−N r N + (r ′ ) −n−2N r 2N + (r ′ ) −n−3N r 3N +··· = M(r ′ )(r ′ ) −n (r/r ′ ) N + (r/r ′ ) 2N + (r/r ′ ) 3N +···

M(r ′ )(r ′ ) −n =

(r ′ /r) N −1

3.2. More about Series 197 We make a digression here, because this is an amazingly good result. The trapezoidal rule

that was used in the calculation of the Taylor coefficients is typically expected to have an error that is O((4θ) 2 ) = O(N −2 ) . (As before, 4θ = 2π/N.) This application is, however,

a very special situation: a periodic analytic function is integrated over a full period . We shall return to results like this in Sec. 5.1.4. In this case, for fixed values of r, r ′ , the truncation error is

O (r/r ′ ) N =O e −η/4θ , η> 0, 4θ →0+. (3.2.19) This tends to zero faster than any power of 4θ !

It follows that a bound for the total error of ˜a n , i.e., the sum of the bounds for the rounding and the truncation errors, is given by

M(r ′ )(r ′ ) −n

U M(r)r +

−n

, r<r < ρ. (3.2.20)

(r ′ /r) N

Example 3.2.2 (Scylla and Charybdis in the Cauchy–FFT). We shall discuss how to choose the parameters r and N so that the absolute error bound of a n , given in (3.2.20) becomes uniformly small for (say) n = 0 : ˆn. 1 + ˆn ≫ 1 is thus the number of Taylor coefficients requested. The parameter r ′ does not belong to the Cauchy–FFT method, but it has to be chosen well in order to make the bound for the truncation error realistic.

The discussion is rather technical, and you may omit it at a first reading. It may, however, be useful to study this example later, because similar technical subproblems occur in many serious discussions of numerical methods that contain parameters that should be appropriately chosen.

First consider the rounding error. By the maximum modulus theorem, M(r) is an increasing function; hence, for r > 1, max n M(r)r −n = M(r) > M(1). On the other hand, for r ≤ 1, max n M(r)r −n = M(r)r − ˆn ; ˆn was introduced in the beginning of this example. Let r ∗

be the value of r, for which this maximum is minimal. Note that r ∗ = 1 unless M ′ (r)/M(r) = ˆn/r for some r ≤ 1. Then try to determine N and r ′ ∈ [r ∗ , ρ) so that, for r = r ∗ , the bound for the second term of (3.2.20) becomes much smaller than the first term, i.e., the truncation error is made negligible compared to the rounding error . This works well if ρ ≫ r ∗ . In such cases, we may therefore choose r = r ∗ , and the total error is then just a little larger than U M(r ∗ )(r ∗ ) − ˆn .

For example, if f (z) = e r , then M(r) = e , ρ = ∞. In this case r ∗ = 1 (since ˆn ≫ 1). Then we shall choose N and r ′

r = N so that e ′ /((r ′ ) N − 1) ≪ eU. One can show that it is sufficient to choose N ≫ | ln U/ ln | ln U||. For instance, if U = 10 −16 , this is

satisfied with a wide margin by N = 32. In IEEE double precision arithmetic, the choice r = 1, N = 32 gave an error less than 2·10 −16 . The results were much worse for r = 10 and

for r = 0.1; the maximum error of the first 32 coefficients became 4 · 10 −4

and 9 · 10 13 (!), respectively. In the latter case the errors of the first eight coefficients did not exceed 10 −10 ,

but the rounding error of a n , due to cancellations, increased rapidly with n. If ρ is not much larger than r ∗ , the procedure described above may lead to a value of N that is much larger than ˆn. In order to avoid this, we set ˆn = αN. We now confine the discussion to the case that r < r ′ <ρ ≤ 1, n = 0 : ˆn. Then, with all other parameters

198 Chapter 3. Series, Operators, and Continued Fractions fixed, the bound in (3.2.20) is maximal for n = ˆn. We simplify this bound; M(r) is replaced

by the larger quantity M(r ′ ) , and the denominator is replaced by (r ′ /r) N . Then, for given r ′ , α, N , we set x = (r/r ′ ) N and determine x so that

M(r ′ )(r ′ ) −αN (U x −α + x)

is minimized . The minimum is obtained for x = (αU) 1/(1+α) , i.e., for r = r ′ x 1/N , and the minimum is equal to 61

M(r ′ )(r ′ ) −n U 1/(1+α) c(α), where c(α) = (1 + α)α −α/(1+α) . We see that the error bound contains the factor U 1/(1+α) . This is proportional to 2 U 1/2

for α = 1, and to 1.65 U 1 for α =

4 . The latter case is thus much more accurate, but for the same ˆn one has to choose N four times as large, which leads to more than four times as

many arithmetic operations. In practice, ˆn is usually given, and the order of magnitude of U can be estimated. Then α is to be chosen to make a compromise between the requirements

for a good accuracy and for a small volume of computation. If ρ is not much larger than r ∗ , we may choose

N = ˆn/α, x = (αU) 1/N 1/(1+α) , r =r ′ x . Experiments were conducted with

f (z) = ln(1 − z),

for which ρ = 1, M(1) = ∞. Take ˆn = 64, U = 10 −15 ,r ′ = 0.999. Then M(r ′ ) =

6.9. For α = 1, 1/2, 1/4, we have N = 64, 128, 256, respectively. The above theory suggests r = 0.764, 0.832, 0.894, respectively. The theoretical estimates of the absolute errors become 10 −9 , 2.4·10 −12 , 2.7·10 −14 , respectively. The smallest errors obtained in experiments with these three values of α are 6·10 −10 , 1.8·10 −12 , 1.8·10 −14 , which were obtained for r = 0.766, 0.838, 0.898, respectively. So, the theoretical predictions of these experimental results are very satisfactory.