Solution of the Approximation Problem

4.5.4 Solution of the Approximation Problem

Orthogonal systems give rise to extraordinary formal simplifications in many situations. We now consider the least squares approximation problem.

Theorem 4.5.13.

If φ 0 ,φ 1 ,...,φ n are linearly independent, then the least squares approximation prob-

= j =0 c j φ j has the unique solution

j =0

The solution is characterized by the orthogonality property that f ∗ − f is orthogonal to all φ j , j = 0 : n. The coefficients c ∗ j are called orthogonal coefficients (or Fourier coefficients ), and satisfy the linear system of equations

called normal equations. In the important special case when φ 0 ,φ 1 ,...,φ n form an or- thogonal system, the coefficients are computed more simply by

(4.5.25) Proof. Let (c 0 ,...,c n )

c j ∗ = (f, φ j )/(φ j ,φ j ), j = 0 : n.

be a sequence of coefficients with c j

∗ j for at least one j. Then

If f ∗ − f is orthogonal to all the φ j , then it is also orthogonal to the linear combination

j =0 (c j −c ∗ j )φ j and, according to the Pythagorean theorem,

Thus if f ∗ − f is orthogonal to all φ k , then f ∗ is a solution to the approximation problem. It remains to show that the orthogonality conditions

c ∗ j φ j − f, φ k = 0, k = 0 : n,

j =0

can be fulfilled. The above conditions are equivalent to the normal equations in (4.5.24). If {φ n

j } j =0 constitutes an orthogonal system, then the system can be solved immediately, since immediately.

4.5. Approximation and Function Spaces 455 Suppose now that we know only that {φ n

j } j =0 are linearly independent. The solution to the normal equations exists and is unique, unless the homogeneous system,

n (φ j ,φ k )c j ∗ = 0, k = 0 : n,

j =0

has a solution c 0 ,c 1 ,...,c n with at least one c i

which contradicts that the φ j were linearly independent. In the case where {φ n j }

j =0 form an orthogonal system, the Fourier coefficients c ∗ j are independent of n (see formula (4.5.25)). This has the important advantage that one can increase the total number of parameters without recalculating any previous ones. Orthogonal

systems are advantageous not only because they greatly simplify calculations; using them, one can often avoid numerical difficulties with roundoff error which may occur when one solves the normal equations for a nonorthogonal set of basis functions.

With every continuous function f one can associate an infinite series,

Such a series is called an orthogonal expansion. For certain orthogonal systems this series converges with very mild restrictions on the function f .

Theorem 4.5.14.

If f ∗ is defined by formulas (

4.5.23) and (4.5.25), then

(c ∗ j ) 2 j 2 .

j =0

Proof. Since f ∗ − f is, according to Theorem 4.5.13, orthogonal to all φ j , 0 ≤ j ≤ n, then f ∗ − f is orthogonal to f ∗ . The theorem then follows directly from Pythagoras’ theorem.

We obtain as corollary Bessel’s inequality:

The series ∞ j =0 (c ∗ j ) 2 j 2 ∗

2 , which is Parseval’s identity.

456Chapter 4. Interpolation and Approximation

Theorem 4.5.15.

If {φ m j } j =0 are linearly independent on the grid G = {x i } i =0 , then the interpolation problem of determining the coefficients {c j m } j =0 such that

has exactly one solution. Interpolation is a special case (n = m) of the method of least m squares. If {φ j } j =0 is an orthogonal system, then the coefficients c j are equal to the orthog- onal coefficients in ( 4.5.25).

Proof. The system of equations (4.5.27) has a unique solution, since its column vectors are the vectors φ j (G) , j = 0 : n, which are linearly independent. For the solution of the

possible seminorm. The remainder of the theorem follows from Theorem 4.5.13. The following collection of important and equivalent properties is named the fun-

damental theorem of orthonormal expansions by Davis [92, Theorem 8.9.1], whom we follow closely at this point.

Theorem 4.5.16.

Let φ 0 ,φ 1 ,φ 2 , · · · be a sequence of orthonormal elements in a complete inner product space H. The following six statements are equivalent: 151

(A) The φ j is a complete orthonormal system in H. (B) The orthonormal expansion of any element y ∈ H converges in norm to y; i.e.,

(C) Parseval’s identity holds for every y ∈ H, i.e.,

|(y, φ j ) | .

j =0

(D) There is no strictly larger orthonormal system containing φ 1 ,φ 2 ,.... (E) If y ∈ H and (y, φ j ) = 0, j = 0, 2, . . . , then y = 0.

(F) An element of H is determined uniquely by its Fourier coefficients, i.e., if (w, φ j ) = (y, φ j ), j = 0, 2, . . . , then w = y. 151 We assume that H is not finite-dimensional, in order to simplify the formulations. Only minor changes are needed in order to cover the finite-dimensional case.

4.5. Approximation and Function Spaces 457 Proof. The proof that A ⇔ B is formulated with respect to the previous statements. By the

conjugations necessary in the handling of complex scalars in inner products (see (4.5.17) and (4.5.18)),

By the Schwarz inequality, (

(x, φ j )(φ j , y) ( (x, φ j ((x, y) − )φ (≤ j (y, φ j )φ j

For the rest of the proof, see Davis [92, pp. 192 ff.].

Theorem 4.5.17.

The converse of statement (F) holds, i.e., let H be a complete inner product space, and let a j

be constants such that ∞ j =0 |a j | 2 < ∞. Then there exists an y ∈ H such that y

= ∞ j =0 a j φ j and (y, φ j ) =a j for all j . Proof. See Davis [92, Sec. 8.9].