Expansions in Orthogonal Polynomials

4.5.6 Expansions in Orthogonal Polynomials

Expansions of functions in terms of orthogonal polynomials are very useful. They are easy to manipulate, have good convergence properties in the continuous case, and usually give

a well-conditioned representation. Let ˆp n denote the polynomial of degree less than n for which

where c j is the jth Fourier coefficient of f and {φ j } are the orthogonal polynomials with respect to the inner product (4.5.30). If we use the weighted Euclidean norm, ˆp n is of course

not a better approximation than p n . In fact,

|f (x) − p n (x) 2 | w(x) dx

|f (x) − ˆ p

n (x) | w(x) dx ≤E n (f )

w(x) dx. (4.5.60)

This can be interpreted as saying that a kind of weighted mean of |f (x) − p n (x) | is less than or equal to E n (f ) , which is about as good a result as one could demand. The error

curve has an oscillatory behavior. In small subintervals, |f (x)−p n (x) | can be significantly greater than E n (f ) . This is usually near the ends of the intervals or in subintervals where

w(x) is relatively small. Note that from (4.5.60) and Weierstrass’ approximation theorem it follows that

for every continuous function f . From (4.5.60) one gets after some calculations ∞

c 2 j 2 n j 2 2,w ≤E n (f ) 2 w(x) dx,

j =n

which gives one an idea of how quickly the terms in the orthogonal expansion decrease.

4.5. Approximation and Function Spaces 467 Example 4.5.11 (Chebyshev Interpolation).

Let p(x) denote the interpolation polynomial in the Chebyshev points x k = cos( 2k+1 π m +1 2 )k = 0 : m. For many reasons it is practical to write this interpolation poly- nomial in the form

p(x) =

c i T i (x).

i =0

Then using the discrete orthogonality property (4.5.51)

The recursion formula (3.2.20) can be used for calculating the orthogonal coefficients ac- cording to (4.5.62).

For computing p(x) with (4.5.61), Clenshaw’s algorithm [71] can be used. Clen- shaw’s algorithm holds for any sum of the form S = n k =1 c k φ k , where {φ k } satisfies

a three-term recurrence relation . It can also be applied to series of Legendre functions, Bessel functions, Coulomb wave functions, etc., because they satisfy recurrence relations of this type, where the α k ,γ k depend on x; see the Handbook [1] or any text on special functions.

Theorem 4.5.21 (Clenshaw’s Algorithm). Suppose that a sequence {p k } satisfies the three-term recurrence relation

(4.5.63) where p −1 = 0. Then

where y n +1 = 0, y n =c n , and y 0 is obtained by the recursion

y k =c k +γ k −1 y k +1 −β k y k +2 , k = n − 1 : −1 : 0.

(4.5.64) Proof. Write the recursion (4.5.63) in matrix form as L n p =p 0 e 1 , where

 L n =   β 1 −γ 1 1  

β n −1 −γ n −1 1

468 Chapter 4. Interpolation and Approximation is unit lower triangular and e 1 is the first column of the unit matrix. Then

n g =g (L T n ) −1 c =g y, where y is the solution to the upper triangular system L T n y = c. Solving this by back-

=c T =c L −1

substitution we get the recursion (4.5.64). It can be proved that Clenshaw’s algorithm is componentwise backward stable with

respect to the data γ k and β k ; see Smoktunowicz [329]. Occasionally one is interested in the partial sums of (4.5.61). For example, if the values of f (x) are afflicted with statistically independent errors with standard deviation σ , then the series can be broken off when for the first time

The proof of Theorem 4.5.19 is constructive and leads to a unique construction of the sequence of orthogonal polynomials φ k , n ≥ 1, with leading coefficients equal to one. The coefficients β k and γ k and the polynomials φ k can be computed in the order

β 0 ,φ 1 (x), β 1 ,γ 0 ,φ 2 (x), . . . .

(Recall that φ −1 = 0 and φ 0 = 1.) This procedure is called the Stieltjes procedure. This assumes that the inner product (xφ k ,φ k )

k 2 can be computed. This clearly can be done for any discrete n-point weight. For a continuous weight function w(x) this is more difficult. One possible way to proceed is then to approximate the integrals using some appropriate quadrature rule. For a discussion of such methods and examples we refer to Gautschi [148].

There are many computational advantages of using the Stieltjes procedure when com- puting the discrete least squares approximation

f (x) ≈p m (x) =

c k φ k (x),

k =0

where φ 0 (x), . . . , φ m (x) are the orthogonal polynomials with respect to the discrete weight function (4.5.32). Recall the family ends with φ m (x) , since

φ m +1 (x) = (x − x 0 )(x −x 1 ) · · · (x − x m )

m +1

natural since there cannot be more than m + 1 orthogonal (or even linearly independent) functions on a grid with m + 1 points.

By Theorem 4.5.13 the coefficients are given by

(4.5.66) Approximations of increasing degree can be recursively generated as follows. Suppose that

c k = (f, φ k )/ k 2 , k = 0 : m.

φ j , j = 0 : k, and the least squares approximation p k of degree k, have been computed. In

4.5. Approximation and Function Spaces 469 the next step the coefficients β k ,γ k −1 , and φ k +1 are computed and the next approximation

is obtained by p k +1 =p k +c k

c k +1 = (f, φ k +1 )/ k +1 . (4.5.67) To compute numerical values of p(x) Clenshaw’s algorithm is used; see Theorem 4.5.21.

+1 φ k +1 ,

For unit weights and a symmetric grid the Stieltjes procedure requires about 4mn flops to compute the orthogonal functions φ k at the grid points. If there are differing weights, then about mn additional operations are needed. Similarly, mn additional operations are required if the grid is not symmetric. If the orthogonal coefficients are determined simultaneously for several functions on the same grid, then only about mn additional operations per function are required. (In the above, we assume m ≫ 1, n ≫ 1.) Hence the procedure is much more

economical than the general methods based on normal equations, which require O(mn 2 ) flops. Since p k is a linear combination of φ j , j = 0 : k, φ k +1 is orthogonal to p k . Therefore, an alternative expression for the new coefficient is

, r k =f−p k , (4.5.68) where r k is the residual. Mathematically the two formulas (4.5.67) and (4.5.68) for c k +1

c k +1 = (r k ,φ k +1 )/ k

are equivalent. In finite precision, as higher-degree polynomials p k +1 are computed, they will gradually lose orthogonality to previously computed p j , j ≤ k. In practice there is an advantage in using (4.5.68) since cancellation will then take place mostly in computing the residual r k =f−p k , and the inner product (r k ,p k +1 ) is computed more accurately.

decreases rapidly with k and then p k provides a good representation of f already for small values of k. Note that for n = m we obtain the (unique) interpolation polynomial for the

given points. When using the first formula one sometimes finds that the residual norm increases when the degree of the approximation is increased ! With the modified formula (4.5.68) this is very unlikely to happen; see Problem 4.5.17. 155

One of the motivations for the method of least squares is that it effectively reduces the influence of random errors in measurements. We now consider some statistical aspects of the method of least squares. First, we need a slightly more general form of Gauss–Markov’s theorem.

Corollary 4.5.22.

Assume that in the linear model ( 1.4.2), ǫ has zero mean and positive definite covari- ance matrix V . Then the normal equations for estimating c are

2 In particular, if V = diag (σ 2

1 ,σ 2 2 ,...,σ n ) this corresponds to a row scaling of the linear model y = Ac + ǫ.

155 The difference between the two variants discussed here is similar to the difference between the so-called classical and modified Gram–Schmidt orthogonalization methods.

470 Chapter 4. Interpolation and Approximation Suppose that the values of a function have been measured in the points x 1 ,x 2 ,...,x m .

Let f (x p )

be the “true” (unknown) function value which is assumed to be the same as the expected value of the measured value. Thus no systematic errors are assumed to be present. Suppose further that the errors in measurement at the various points are statistically independent . Then we have a linear model f (x p ) =

be the measured value, and let ¯ f (x p )

¯ f (x p ) + ǫ, where E(ǫ) = 0,

1 ,...,σ 2 n ). (4.5.70) The problem is to use the measured data to estimate the coefficients in the series

c j φ (x), n ≤ m,

j =1

where φ 1 ,φ 2 ,...,φ n are known functions. According to the Gauss–Markov theorem and its corollary the estimates c j ∗ , which one gets by minimizing the sum

have a smaller variance than the values one gets by any other linear unbiased estimation method. This minimum property holds not only for the estimates of the coefficients c j , but also for every linear functional of the coefficients, for example, the estimate

of the value f (α) at an arbitrary point α. Suppose now that σ p = σ for all p and that the functions {φ j n } j =1 form an orthonormal system with respect to the discrete inner product

(f, g) =

f (x p )g(x p ).

p =1

Then the least squares estimates are c ∗ j = (f, φ j ) , j = 1 : n. According to (1.4.4) the estimates c ∗ j and c k ∗

k − ¯c k ) }= σ 2 if j = k. From this it follows that

E {(c ∗ j − ¯c j )(c ∗

}|φ 2 j (α) 2 2 | =σ

var{f (α) } = var

c ∗ j φ j (α) =

var{c j ∗

As an average, taken over the grid of measurement points, the variance of the smoothed function values is

|φ j (x i ) 2 | 2 =σ . m j =1

var{f n (x i ) }=

m j =1 i =1

4.5. Approximation and Function Spaces 471 Between the grid points, however, the variance can in many cases be significantly larger.

For example, when fitting a polynomial to measurements in equidistant points, the Gram polynomial P n,m can be much larger between the grid points when n > 2m 1/2 . Set

I =σ

|φ(x)| dx.

j =1 2 −1

Thus σ 2 I is an average variance for f n ∗ (x) taken over the entire interval [−1, 1]. The following values for the ratio ρ between σ 2 I and σ 2 (n + 1)/(m + 1) when m = 42 were obtained by H. Björk [32]. Related results are found in Reichel [298].

n +1 5 10 15 20

3 7 1.0 1.1 1.7 26 7 · 10 11 1.7 · 10 8 · 10 These results are related to the recommendation that one should choose n < 2m 1/2 when

fitting a polynomial to equidistant data. This recommendation seems to contradict the Gauss–Markov theorem, but in fact it just means that one gives up the requirement that the estimates are unbiased. Still, it is remarkable that this can lead to such a drastic reduction of the variance of the estimates f n ∗ .

If the measurement points are the Chebyshev abscissae, then no difficulties arise in fitting polynomials to data. The Chebyshev polynomials have a magnitude between grid points not much larger than their magnitude at the grid points. The average variance for f n ∗

becomes the same on the interval [−1, 1] as on the net of measurements, σ 2 (n + 1)/(m + 1). The choice of n when m is given is a question of compromising between taking into

account the truncation error (which decreases as n increases) and the random errors (which grow when n increases). If f is a sufficiently smooth function, then in the Chebyshev case |c j | decreases quickly with j. In contrast, the part of c j which comes from errors in measurements varies randomly with a magnitude of about σ (2/(m + 1) 1/2 ) , using (4.5.50)

j 2 = (m + 1)/2. The expansion should be broken off when the coefficients begin to “behave randomly.” An expansion in terms of Chebyshev polynomials can hence be used for filtering away the “noise” from the signal, even when σ is initially unknown.

Example 4.5.12.

Fifty-one equidistant values of a certain analytic function were rounded to four dec- imals. In Figure 4.5.2, a semilog diagram is given which shows how |c i | varies in an

expansion in terms of the Chebyshev polynomials for these data. For i > 20 (approxi- mately) the contribution due to noise dominates the contribution due to signal. Thus it is sufficient to break off the series at n = 20.