The Singular Value Decomposition

1.4.3 The Singular Value Decomposition

In the past the conventional way to determine the rank of a matrix A was to compute the row echelon form by Gaussian elimination. This would also show whether a given linear system is consistent or not. However, in floating-point calculations it is difficult to decide if a pivot element, or an element in the transformed right-hand side, should be considered as zero or nonzero. Such questions can be answered in a more satisfactory way by using the singular value decomposition (SVD), which is of great theoretical and computational importance. 16

The geometrical significance of the SVD is as follows: The rectangular matrix A ∈ R m ×n

, m ≥ n, represents a mapping y = Ax from R n to R m . The image of the unit sphere

2 m = 1 is a hyper ellipse in R with axes equal to σ 1 ≥σ 2 ···≥σ n ≥ 0. In other words, the SVD gives orthogonal bases in these two spaces such that the mapping is represented

by the generalized diagonal matrix H ∈ R m ×n . This is made more precise in the following theorem, a constructive proof of which will be given in Volume II.

Theorem 1.4.4 (Singular Value Decomposition). Any matrix A ∈ R m ×n of rank r can be decomposed as

(1.4.15) where H r = diag (σ 1 ,σ 2 ,...,σ r ) is diagonal and

A = UHV T ,

0 0 ∈R ×n ,

= (u n 1 ,...,u m ) ∈R ×m , V = (v 1 ,...,v n ) ∈R ×n (1.4.16)

16 The SVD was published by Eugenio Beltrami in 1873 and independently by Camille Jordan in 1874. Its use in numerical computations is much more recent, since a stable algorithm for computing the SVD did not become

available until the publication of Golub and Reinsch [167] in 1970.

1.4. The Linear Least Squares Problem

are square orthogonal matrices, U T U =I m ,V T V =I n . Here σ 1 ≥σ 2 ≥···≥σ r > 0

are the r ≤ min(m, n) nonzero singular values of A. The vectors u i , i = 1 : m, and v j , j = 1 : n, are left and right singular vectors. (Note that if r = n and/or r = m, some of the zero submatrices in H disappear.)

The singular values of A are uniquely determined. For any distinct singular value σ j

j is unique (up to a factor ±1). For

a singular value of multiplicity p the corresponding singular vectors can be chosen as any orthonormal basis for the unique subspace of dimension p that they span. Once the singular vectors v j , 1 ≤ j ≤ r, have been chosen, the vectors u j , 1 ≤ j ≤ r, are uniquely determined, and vice versa, by

Av j ,

A T u j , j = 1 : r. (1.4.17)

By transposing (1.4.15) we obtain A T =VH T U T , which is the SVD of A T . Ex- panding (1.4.15), the SVD of the matrix A can be written as a sum of r matrices of rank

The SVD gives orthogonal bases for the range and null space of A and A T . Suppose that the matrix A has rank r < min(m, n). It is easy to verify that

R(A) = span (u 1 ,...,u r ),

N (A T ) = span (u r +1 ,...,u m ), (1.4.19)

N (A) = span (v r +1 ,...,v n ). (1.4.20) It immediately follows that R(A) ⊥

R(A T ) = span (v 1 ,...,v r ),

= R(A T ) ; (1.4.21) i.e., N (A T ) is the orthogonal complement to R(A), and N (A) is the orthogonal complement

= N (A T ),

N (A) ⊥

to R(A T ) . This result is sometimes called the fundamental theorem of linear lgebra. We remark that the SVD generalizes readily to complex matrices. The SVD of a matrix A ∈ C m ×n is

2 where the singular values σ 1 ,σ 2 ,...,σ r are real and nonnegative, and U and V are square

unitary matrices, U H U =I m ,V H V =I n . (Here A H denotes the conjugate transpose of A.) The SVD can also be used to solve linear least squares problems in the case when the

the least squares solution is not unique, then there exists a unique least squares solution of minimum Euclidean length , which solves the least squares problem

min

= {x ∈ R

2 = min}. (1.4.23)

∈S

52 Chapter 1. Principles of Numerical Calculations In terms of the SVD (1.4.22) of A the solution to (1.4.23) can be written x = A † b , where

the matrix A † is

H −1 0

0 0 2 ∈R The matrix A †

is unique and called the pseudoinverse of A, and x = A † b is the pseudoin- verse solution. Note that problem (1.4.23) includes as special cases the solution of both

overdetermined and underdetermined linear systems. The pseudoinverse A † is often called the Moore–Penrose inverse. Moore developed the concept of the general reciprocal in 1920. In 1955 Penrose [288] gave an elegant algebraic characterization and showed that X = A † is uniquely determined by the four Penrose conditions :

( 1) AXA = A, ( 2) XAX = X, (1.4.25) ( 3) (AX) T = AX,

( 4) (XA) T = XA. (1.4.26) It can be directly verified that X = A † given by (1.4.24) satisfies these four conditions. In particular this shows that A † does not depend on the particular choices of U and V in the

SVD. The orthogonal projections onto the four fundamental subspaces of A have the fol- lowing simple expressions in terms of the SVD:

R(A) T = AA =U 1 U 1 , P N (A T ) = I − AA =U 2 U 2 , (1.4.27) P R(A T

=V T 1 V 1 , P N (A) =I−A A =V 2 V 2 . These first expressions are easily verified using the definition of an orthogonal projection

) =A A T

and the Penrose conditions.