Inner Product Spaces and Orthogonal Systems

4.5.3 Inner Product Spaces and Orthogonal Systems

An abstract foundation for least squares approximation is furnished by the theory of inner product spaces which we now introduce.

4.5. Approximation and Function Spaces 451

Definition 4.5.8.

A normed linear space S will be called an inner product space if for each two elements

f, g in S there is a scalar designated by (f, g) with the following properties:

1. (f + g, h) = (f, h) + (g, h) (linearity),

2. (f, g) = (g, f ) (symmetry),

3. (f, αg) = α(f, g), α scalar (homogeneity),

(positivity). The inner product (f, g) is scalar, i.e., real in a real space and complex in a complex space.

is a seminorm on S. We shall show below that the triangle inequality is satisfied. (The other axioms for a

norm are obvious.) The standard vector inner products introduced in Sec. A.1.2 in Online Appendix A are particular cases, if we set (x, y) = y T x in R n

and (x, y) = y H x in C n .A complete inner-product space is called a Hilbert space and is often denoted H in this book.

One can make computations using the more general definition of (f, g) given above in the same way that one does with scalar products in linear algebra. Note, however, the conjugations necessary in a complex space,

(4.5.17) because, by the axioms,

(αf, g) = ¯α(f, g),

(αf, g) = (g, αf ) = α(g, f ) = ¯α(g, f ) = ¯α(f, g).

By the axioms it follows by induction that

(φ k ,c j φ j ) =

c j (φ k ,φ j ). (4.5.18)

Theorem 4.5.9 (Cauchy–Schwarz inequality). The Cauchy–Schwarz inequality in a complex space is

Proof. Let f , g be two arbitrary elements in an inner-product space. Then, 150 for every real number λ,

2 2 0 ≤ (f + λ(f, g)g, f + λ(f, g)g) = (f, f ) + 2λ|(f, g)| 2 +λ |(f, g)| (g, g). This polynomial in λ with real coefficients cannot have two distinct zeros, hence the dis-

criminant cannot be positive, i.e.,

4 |(f, g)| 2 − (f, f )|(f, g)| (g, g) ≤ 0.

So, even if (f, g) = 0, |(f, g)| 2 ≤ (f, f )(g, g).

150 We found this proof in [304, sec. 83]. The application of the same idea in a real space can be made simpler.

452 Chapter 4. Interpolation and Approximation By Definition 4.5.8 and the Cauchy–Schwarz inequality,

2 = (f + g, f + g) = (f, f ) + (f, g) + (g, f ) + (g, g)

2 2 2 . This shows that the triangle inequality is satisfied by the norm defined above.

Example 4.5.9.

i =1 |x i | < ∞ and which is equipped with the inner product

The set of all complex infinite sequences {x 2 i } for which

(x, y) =

i =1

constitutes a Hilbert space.

Definition 4.5.10.

Two functions f and g are said to be orthogonal if (f, g) = 0. A finite or infinite sequence of functions φ 0 ,φ 1 ,...,φ n constitutes an orthogonal system if

(φ i ,φ j )

Theorem 4.5.11 (Pythagoras’ Theorem). Let {φ 1 ,φ 2 ,...,φ n } be an orthogonal system in an inner-product space. Then

The elements of an orthogonal system are linearly independent. Proof. We start as in the proof of the triangle inequality:

2 2 . Using this result and induction the first statement follows. The second statement then

follows because

c j φ j = 0 ⇒ |c j | = 0 for all j.

Theorem 4.5.12.

A linear operator P is called idempotent if P = P 2 . Let V be the range of P . Then P is a projection (or projector) onto V if and only if P is idempotent and P v = v for each

v ∈ V.

Proof.

If P is a projection, then v = P x for some x ∈ B, hence P v = P 2 x = P x = v. Conversely, if Q is a linear operator such that Qx ∈ V for all x ∈ B, and v = Qv for all

v ∈ V , then Q is a projection; in fact Q = P .

4.5. Approximation and Function Spaces 453 Note that I − P is also a projection, because

(I − P )(I − P ) = I − 2P + P 2 =I−P. Any vector x ∈ B can be written uniquely in the form

(4.5.20) Important examples of projections in function spaces are interpolation operators, for

x = u + w, u = P x, w = (I − P )x.

example, the mapping of C[a, b] into P k by Newton or Lagrange interpolation, because each polynomial is mapped to itself. The two types of interpolation are the same projection,

although they use different bases in P k . Another example is the mapping of a linear space of functions, analytic on the unit circle, into P k so that each function is mapped to its Maclaurin expansion truncated to P k . There are analogous projections where periodic functions and trigonometric polynomials are involved

In an inner-product space, the adjoint operator A ∗ of a linear operator A is defined by the requirement that

(A ∗ u, v) = (u, Av) ∀ u, v.

An operator A is called self-adjoint if A = A T ∗ . In R v , i.e., the standard scalar product. Then (A ∗ u) T

, we define (u, v) = u

T Av , i.e., u T ((A ∗ ) =u T v =u T Av , hence

A ∗ =A T . It follows that symmetric matrices are self-adjoint in R n . In C n

, with the inner product (u, v) = u H v , it follows that A ∗ =A H , i.e., Hermitian

Example 4.5.10.

An important example of an orthogonal system is the sequence of trigonometric func- tions φ j (x) = cos jx, j = 0 : N − 1. These form an orthogonal system, with either of the two inner products

(continuous case),   0

f (x)g(x) dx

(f, g) = N (4.5.22)  −1

  f (x i )g(x i ), x i

2i + 1 π

(discrete case).

i =0

Moreover, it holds that 

 1   π, j

(continuous case), j 2 = 2    1 N, j

0 = N (discrete case).

2 These results are closely related to the orthogonality of the Chebyshev polynomials; see

Theorem 4.5.20. Trigonometric interpolation and Fourier analysis will be treated in Sec. 4.6. There are many other examples of orthogonal systems. Orthogonal systems of poly-

nomials play an important role in approximation and numerical integration. Orthogonal systems also occur in a natural way in connection with eigenvalue problems for differential equations, which are quite common in mathematical physics.

454 Chapter 4. Interpolation and Approximation